Improvements to DSDL Container Build Process

We’re happy to announce that with the upcoming release of Splunk App for Data Science and Deep Learning (DSDL) 5.1.1 we’re significantly overhauling the build process for containers in DSDL.

More and more customers are adopting DSDL for some of their most complex and advanced workloads. 

In this newest release, we’re making the process of deploying, building and maintaining containers for DSDL more modular, more secure, more robust, and more scalable as well as adding some new features!

More modular:

• We’ve completely overhauled the build scripts and Dockerfiles to make it easy to combine sets of requirements, base images, and implementation details into various combinations so that you can easily build or modify containers appropriate for your environment and security constraints.

More secure:

• We now provide a set of base images derived from the redhat UBI9 container image. This means that many more customers who were not able to deploy Ubuntu-based images will be able to do so, or will have templates to build their own variations with ease.
• For simple development environments, self-signed certificates are now generated automatically at build time making them more secure when deployed quickly, and it's easy to add custom certificates in the build process by adding them to the certificates directory.
• We’ve also added a script to make it easy to scan and report on new containers with Trivy and get vulnerability reports easily as part of the build process.

More robust:

• We’ve added a testing framework (pytest and playwright) to the build scripts to make testing new combinations of requirements for basic functionality much easier. We’ll be looking to significantly expand our testing over time as functionality continues to improve and increase.

New features:

• We’ve added a beta template for deploying DSDL with docker-compose rather than relying on a connection between DSDL and the Docker control API. In some environments control API connectivity is not possible due to security constraints and we’re planning to expand this passive deployment approach for K8s and HELM in the near future.
• We’ve included a new image configuration which contains all of the pre-built content from our amazing threat research team making it easier than ever to get pre-packaged advanced content deployed with DSDL ES and ESCU.
• We’ve added some content and examples around using DSDL to connect to a containerised vector database (Milvus) which can also be deployed with docker-compose. This works well with some text-vectorization examples which are also new for this version.

Deploying advanced machine learning content into a Security or ITOps environment can be a daunting process but we hope with these updates you find it easier than ever to lead the industry and get ahead of your adversaries and issues.