Machine Learning at Splunk in Just a Few Clicks

The Machine Learning team at Splunk has been hard at work over the last several months preparing for a few exciting launches at .conf22, held just a few weeks ago. Splunk customers want to leverage machine learning (ML) in their environments, but many aren’t sure how to use it, or even how to get started.

The Splunk Machine Learning Toolkit (MLTK) is very powerful, offering both guided and custom approaches to ML, and has been immensely popular with our customers, being one of the most-downloaded apps from Splunkbase. Still, many users are just looking for one-click-style experiences, where they can harness the promise of ML to simplify tasks that were previously complex and time consuming without performing any of the rituals associated with operationalization.

To meet this growing requirement, we are pleased to release three new apps in beta from Splunk Works: Anomaly Detection Assistant for Splunk (beta), Smart Alerts Assistant for Splunk (beta), and SPL Copilot for Splunk (beta). These applications are designed to make ML even more accessible to Splunk customers. This blog will walk through each of the applications in brief, providing an overview of how they might be utilized.

Anomaly Detection

Shown in the .conf22 keynote address as well as the Platform Super Session, Anomaly Detection Assistant for Splunk (beta) removes the guesswork when trying to find anomalies in time-series datasets and identifies them in just a few clicks for rapid time to insight. 

One common approach to finding anomalies in time-series data is to use basic summary statistics (e.g. mean and standard deviation) over sliding windows, as shown in our documentation.

The user is tasked with finding values for parameters such as the window length and standard deviation multiplier (in the query above, 100 and 2 respectively). Users have to do this by manually changing the parameters and inspecting the results, rinsing and repeating until they are satisfied with the detected anomalies. 

With the Anomaly Detection Assistant for Splunk (beta), ML will automatically generate the optimal SPL query. Users can simply identify the time-series dataset they want to identify anomalies in and the app will not only identify the anomalies in a convenient visual, but also will present the auto-tuned query by clicking “Show SPL.”

This query can then be used anywhere in the Splunk environment. Anomaly detection in just a couple of clicks!

Smart Alerts

Smart Alerts Assistant for Splunk (beta) enables customers to focus on issues that matter the most, instead of spending time triaging alerts. The assistant produces a recommended, stack-ranked priority for each alert, and can improve the ranking over time. 

Each of these applications, including Smart Alerts, comes with pre-populated sample data to help users get started. To use Smart Alerts with a custom alert dataset, simply input the dataset in the initial search. After confirming the dataset and choosing “Rank Alerts,” the Prioritized Alerts section populates with the initial ranking.

The “Rank” column indicates the importance of the alerts according to our ML framework. As one might expect, alerts at the top of the ranking are more important than alerts lower in the ranking. 

The core functionality of the Smart Alerts Assistant is that it is able to learn from a user's interactions with its rankings about which kinds of alerts the user is most interested in. Over time, rankings become better-tailored to meet the user's individual needs. Expanding an alert in the UI signals to the application that the user is interested in this alert and those like it. Similarly, marking an alert as a “False Positive” signals that this alert, and alerts like it, should be lower in priority. 

How does this work? Each of the user interactions — alert expansions or marking as false positives — triggers a retraining of the alert ranking model. We even show this action at the bottom of this screen so the user can see what’s happening.

Over time, the model learns from the user’s behavior, interest, and inputs, producing increasingly relevant and accurate alert ranking.

Natural-Language Querying

The Splunk Search Processing Language (SPL) is incredibly powerful, and many customers — new and intermediate users alike — reference the documentation frequently. But what if there were a quicker way to get results from the data?

SPL Copilot for Splunk (beta) empowers users to search their data using plain English. Now customers can write a description of what they want in plain English, and Copilot translates the request into query ideas that can be executed or built upon, all within a familiar Splunk interface. Under the hood, SPL Copilot for Splunk uses machine learning to translate plain English instructions into executable SPL query ideas. 

SPL Copilot for Splunk (beta) also comes with a pre-populated sample data index to get started, or users can select their own index if they’re ready. Even the process of writing the input SPL has been simplified with a simple dropdown.

Next, users can choose from a selection of example “plain English” queries, or they can write their own using the examples as inspiration. Clicking “Translate to SPL” does just that! The app returns a few suggested SPL queries based on the plain English input, and ranks them in order of probability that they address the query.

Each of the suggested SPL queries can be opened directly in search, or can be further customized right inside the app by clicking the query and adjusting it in the box below.

SPL Copilot for Splunk makes SPL more accessible and helps customers answer questions about their data faster. Interested in a more technical walkthrough of how SPL Copilot for Splunk (beta) works? Check out our blog post! 

Wrap-Up

If you missed us at the Machine Learning booth at .conf22 in Las Vegas this year and didn’t get a chance to see the apps live, make sure to give them a try. The applications are available for everyone on Splunkbase – download them today.

• Anomaly Detection Assistant for Splunk (beta) - Remove the guesswork when trying to find anomalies in time-series datasets and identify them in just a few clicks for rapid time to insight.
• Smart Alerts Assistant for Splunk (beta) - Focus on issues that matter the most, instead of spending time triaging alerts, with a recommended, stack-ranked priority for each alert.
• SPL Copilot for Splunk (beta) - Empower Splunk users and answer questions faster with the ability to search data using normal English! Simply write a description of what you want in English, and Copilot translates the request into SPL query ideas.

We are excited to share these latest developments from the ML team, and look forward to feedback from our customers and partners. Make sure to keep an eye on the official .conf website for additional ML content and breakout session replays.

For those who want to go deeper into what is possible with ML at Splunk — or get some inspiration into use cases that can be enabled with ML — please check out some of our webinars, such as how to prevent data downtime with machine learning. Alternatively, we have also just released a set of ML deep dives that provide end-to-end guides for how to implement some common use cases with MLTK. 

Happy Splunking!

Special thanks to Abraham Starosta (Senior Applied Scientist), Kristal Curtis (Engineering Manager), and Julien Veron Vialard (Applied Scientist) for their contributions to this blog.