Learn SPL command types: Efficient search execution order and how to investigate them

When performing searches, Splunk uses its own language, SPL (Search Processing Language).

SPL commands can be categorized into several types depending on the processing they perform. Especially in a distributed environment where the Splunk system is made up of multiple servers, if you do not understand which components perform heavy processing depending on the SPL type, you may create inefficient searches.

In this article, we will explain each type of SPL and show you the efficient order in which to run searches and how to use the Search Job Inspector, an investigative tool.

Streaming and Non-Streaming Commands

SPL types can be broadly divided into two types: Streaming commands and Non-Streaming commands. These can then be further divided into six types. As follows:

• Streaming
  • Distributable Streaming
  • Centralized Streaming
• Non-Streaming
  • Transforming
  • Generating
  • Orchestrating
  • Dataset processing
    

First, we will explain the difference between Streaming commands and Non-Streaming commands.

1-1. Streaming command

Streaming commands are commands that operate on data on an individual event basis. For example, the eval and head commands are Streaming commands.

These commands can basically be described as commands that take one piece of data (event) as input and output one piece of data (event) in response. If ordering is not a concern, you can process events one by one without waiting for all events to arrive. This is a key feature of the Streaming command.

These Streaming commands can be further divided into Distributable Streaming commands, which do not need to consider event ordering, and Centralized Streaming commands, which need to consider data ordering.

The most important type is the Distributable Streaming command. Commands classified as this type allow for efficient searches because they can perform distributed processing on the Indexer side. I'll explain this in more detail later.

1-2. Non-Streaming Commands

In contrast, Non-Streaming commands are, as the name suggests, commands other than Streaming commands.

It is called this way to distinguish it from the Streaming command. For more details, see the Transforming, Generating, Orchestrating and Dataset processing command descriptions.

Detailed command type description

Next we'll explain each command type in more detail.

2-1. Distributable Streaming command

Among the Streaming commands, those that do not need to take into account the event order are called Distributable Streaming commands. Examples include eval, rex, and rename. This command allows you to immediately process each event retrieved from multiple indexers individually when you run a search.

For this reason, in an environment consisting of multiple indexers in a distributed configuration, searches are performed in such a way that each indexer performs parallel processing, and then the results are aggregated by the search head and displayed to the user. (It does not necessarily have to be processed by the Indexer. If the search results have already been aggregated by the Search Head, the Distributable Streaming command will also be processed by the Search Head.)

The Distributable Streaming command's unique feature and greatest advantage is that it allows processing to be distributed by the Indexer. The list of Distributable Streaming commands is below. If you use these commands when writing SPL, try to place them at the beginning of the SPL as much as possible so that the Indexer can distribute the processing.

• Streaming commands (See the commands with "Distributed streaming" in the "Notes" column)

2-2. Centralized Streaming Command

Among the streaming commands, those that require consideration of event ordering are called centralized streaming commands. Examples include head and transaction.

Unlike the Distributable Streaming command, this command requires that the order is important, so events must be aggregated in the Search Head before being processed. Because the Search Head needs to perform the processing, parallel processing like the Distributable Streaming command is not possible, even in a distributed environment.

2-3. Transforming command

Commands that transform input data to produce a new data set are called Transforming commands. Examples include stats, charts, and tables. When you use these commands, the events are converted into a data table by converting their data structure. Data in the form of a data table is useful for visualization.

Like the Centralized Streaming commands, these commands cannot be processed in parallel and events must be aggregated in the Search Head before being processed.

2-4. Generating command

A command that generates new data is called a Generating command. Examples include search, rest, and makeresults. The word "search" omitted at the beginning of the search command is actually a Generating command. These commands are usually preceded by a pipe (|) and run as "| rest <URL>" and so on. These commands are executed from the Search Head.

2-5. Orchestrating commands

The commands that manage the entire search process are called Orchestrating commands. Examples include require and localop. These commands manage the search process that Splunk runs and do not affect the results of your searches. These commands are executed from the Search Head.

2-6. Dataset Processing Commands

Commands that process search results are called Dataset Processing commands. Examples include sort, tail, and fillnull.

These commands do things like sorting and filtering your search results. For example, the sort and dedup commands sort and remove duplicates from a dataset. These commands are executed from the Search Head.

Considerations for performing efficient searches

For example, consider the following search:

[Case 1]

index=main sourcetype=access_combined_wcookie
| stats count by clientip
| rename clientip as sourceIp
| table sourceIp, count

In fact, the above search may be inefficient in a distributed environment.The reason for this is that the Distributable Streaming command rename is used after the Transforming command stats.

The Distributable Streaming command allows distributed processing on the Indexer. However, in the above example, the Transforming command (stats) is used first. In this case, when applying stats, all events will be aggregated at the search head side. As a result, the rename process is executed by the Search Head, which means that parallel processing is not possible and the Search Head has to perform the process alone.

(However, if the number of events after running the stats command is very small, it may be more efficient to run the stats command and then run the rename. In this example, we will assume that there are a large number of client IPs and the rename process takes time.)

To process Distributable Streaming commands in parallel using the Indexer, change the order as follows:

[Case 2]

index=main sourcetype=access_combined_wcookie
| rename clientip as sourceIp
| stats count by sourceIp
| table sourceIp, count

This order allows rename commands to be processed in parallel by the Indexer, and then the Search Head aggregates the events and processes the stats.

Distributable Streaming commands should be used at the beginning of an SPL whenever possible.

Inspecting Search Jobs with Search Job Inspector

A useful tool for inspecting search execution is the Search Job Inspector. This time, we will use this tool to investigate the results of search job execution. The Search Job Inspector can be launched by selecting "Jobs" - "Inspect Jobs" after running a search.

Alternatively, you can start the process by selecting "Activity" - "Jobs" from the menu bar, and then selecting "Actions" - "Jobs" - "Investigate Job" for the job you want to investigate.

[Starting from "Jobs" after performing a search]

When you launch Search Job Inspector, the search results will be displayed at the top.This bar shows you the number of results your search returned, the number of events scanned when you ran the search, and the time it took.

Search job results for [Case 1]

Search job results for [Case 2]

As you can see from the above, [Search 1] took 1.6 times longer. (However, search times may vary slightly depending on execution timing.)

The Search Job Inspector also gives you other information that can help you make your search jobs more efficient. By looking at the "Search Job Properties" in the search job results, you can see how Splunk interpreted the SPL you specified and the actual search that was performed.

Search job properties for [Case 1]

Search job properties for [Case 2]

remoteSearch represents a search performed on a remote server, i.e. the Indexer, from the perspective of the Search Head. reportSearch is a search performed by the Search Head itself. As can be seen above, in Case 1, the Search Head performs the process of “renaming clientip as sourceIp”, which indicates that the amount of processing performed by the Search Head is large.

In addition, the Search Job Inspector also has a feature that allows you to visually see which components are taking how long to process when you run a search job. This is a useful tool if a specific search process is taking a long time and you want to consider improving it, so be sure to give it a try.

Summary

This time, we introduced the types of Splunk SPL commands and the Search Job Inspector, a tool for actually inspecting the results of search jobs.

To summarize the important points for making searches more efficient, if there is a Distributable Streaming command, execute it at the top of the SPL as much as possible and make arrangements for parallel processing on the Indexer side, and if necessary, use a tool such as Search Job Inspector to investigate the job execution status.

If you are interested, please read the reference links below to learn more about Splunk command types and how to use Search Job Inspector.

• Types of commands
• Command types
• Write better searches
• View search job properties