Scaling Anomaly Detection with MLTK 5.5

Who doesn’t love a bit of anomaly detection with Splunk? As someone who has spent far too long talking about cyclical statistical forecasts and anomalies, you’ll be relieved that this is a topic that we don’t get tired of here at Splunk! In this blog post we will be taking you through some of the recent changes to the Machine Learning Toolkit, where we have released a more scalable version of our users most favorite algorithm.

Scaled Anomaly Detection

The recently released Machine Learning Toolkit (MLTK) 5.5 introduces a new version of our most popular algorithm for detecting anomalies: the DensityFunction algorithm. This algorithm was immediately in demand on release, with some awesome conf talks showing how it was adopted, such as this one from StubHub. More recently we have seen examples where this algorithm has been used to monitor data ingest pipelines into Splunk as well as for Know Your Customer (KYC) use cases.

Despite the popularity of this algorithm, however, we have had consistent feedback that it can be difficult to apply it to large datasets. So in this new release we have given you a new scaled version of the algorithm, which in some cases provides an 8-9 times improvement on the previous version of the algorithm!

How Can I Use It?

Well that is super simple! You can either use it directly in the search using the new supervise_split_by option and setting it to true as per the search snippet below, otherwise you can use it in the Smart Outlier Detection Assistant as shown in the screenshot.

... | fit DensityFunction count by "app,day_of_week"
supervise_split_by=true ...

If you want to know more about how this works and understand the changes we made to the algorithm better please refer to this lantern article or check out this video.

Where Can I Get Started?

First up, go and download MLTK 5.5 and try out this new approach for yourself! If you want some inspiration on some use cases to try out it is also worth checking out our security use cases enhanced by AI and ML or an introductory use case guide: Splunk AI for observability.

Coming up next for MLTK we are exploring integrations with LLMs so that you can utilise your own LLM inside of a Splunk search. If you want to know more about this - or even try it out yourself - head on over to the preview portal to sign up for taking part!

Happy Splunking!