Admission Rules: Use Cases and Best Practices

We are expanding the capabilities of Splunk Workload Management by introducing admission rules that you may leverage to control searches that run on Splunk. Poorly constructed searches impacting high priority workloads is a common occurrence and hard to prevent. With admission rules, you can automatically filter such searches and provide users a contextual message to improve their query. In this blog, I will describe a few use cases and offer some guidance on best practices for using admission rules.

Use Case 1: Filtering Poorly Constructed Searches

Although different Splunk admins may have different definitions of what construes a poorly written search, there are a few general rules. For example, wild card searches can generally have a big impact unless there are very few indexes and the search is done for a short duration. On the other hand, you may also want to restrict all time duration searches triggered by users and applications.

Use Case 2: Limiting Searches During Peak Time

You may want to limit certain users or types of searches from running during peak hours. For example, new users may not be allowed to run ad hoc searches during peak hours or certain groups of users may not be allowed to run real-time searches.

To sum up, admission rules provide you a rich framework using the same predicates as workload management — role, user, search type, search mode, index and search time duration — to control which types of searches, from which users are allowed during a certain period. A few examples of the rules that can be considered are below:

The admission rules are a powerful tool, hence, they should be used carefully. There are a few best practices that you should follow:

Reduce the Surface Area While Creating a Rule

There are several rules that may seem like a no brainer to implement at first but may have unintended consequences. For example, if you create a rule to filter all index=* searches, you may stop several data model acceleration (DMA) searches that by default use index=*. Or if you create a rule to filter all-time searches, you may impact the Splunk monitoring app that uses all time searches.

A better way to create admission rules is to be more specific. For example, creating a rule that prevents all non-admin roles from running an ad hoc or scheduled search that uses index=*

index=* AND (NOT role=*admin) AND (search_type=adhoc OR search_type=scheduled)

Monitor the Impact After Creating a Rule

It is important to ensure that the rules that you created are not causing unintended consequences. Hence, it is advisable to create one rule at a time and monitor the impact of the rule for at least 24 hours before adding more rules. Since admission rules filter the searches, you may monitor total search count over time and account for any drops. For example, the search shown below may be used to see which admission rules are getting triggered and which user is getting impacted.

index=_internal source=*wlm_monitor.log prefilter_action=filter | stats count by prefilter_rule user

As we look into the future, we will extend the capabilities of admission rules such that it becomes a central place for admins to manage search admission on Splunk. Please check out the Splunk App showcase video in the meantime on admission rules.