Implementation of Incentive Driven User Access

Out of the box, a Splunk user has the capabilities to do some powerful stuff – but as Uncle Ben tells us, “with great power comes great responsibility“. In my prior post, we reviewed the scenario and purpose behind Incentive Driven User Access. In a this post, we’ll dive into the conf files and explore what settings are worth reviewing to implement such a solution.

Authentorizationing….?!

Let’s conceptually differentiate the settings for authorization and those of authentication. The names are so darn similar that without understanding their differences, you’re bound to mix them up.

When you first navigate to your Splunk deployment, you need to prove that you’re a valid user. To do this, Splunk will need to verify that the username and password you provide are valid, or authentic. Splunk does this by checking your credentials against a user repository (local, ldap, etc…). This process, of checking if you’re authentic, is an authentication process. As such, the details that inform Splunk how and where to check the credentials are stored in a configuration file, authentication.conf.

After Splunk knows it’s ok to let you in, it needs to know what things you can do. Are you allowed to run searches? View apps? Explore data? For all of these activities, and more, Splunk must determine if you’re authorized or not. As such, the details that inform Splunk what you’re allowed (or not allowed) to do and see are stored in a configuration file, authorize.conf.

authentication.conf = user validation definitionauthorization.conf = feature and data access definition

Since our official Splunk documentation describes step by step ways of using these files, consider the above description an easy access map in case you get lost through this reading.

All Your Authorize Belongs To Us

With that all squared away, let’s jump into some specific authorize.conf attributes that might be helpful here. Remember that the focus is on what items can be used as rewards for demonstrating Splunk proficiency. In other words, focus on how the absence or limitation of a given feature can be used to incentivize a user into learning more Splunk and demonstrating the proficiencies appropriate for using that feature responsibly.

Capabilities

When it comes to granting capabilities, I delineate what features to give to uneducated users as compared to educated power users with this motive:

Will this feature impact the Splunk deployment when the user is NOT logged in?

For example, if an uneducated user is able to create a mal-formed realtime search, and schedule it to run all time – that’s very impacting to the environment and I want to limit or mitigate that from occurring.

With that in mind, here are some capabilities worth tweaking, broken out by reason. This is not a review of every capability since out-of-the-box, Splunk already provides guidance on many (example: delete_by_keyword). Refer to the List of available capabilities for explanations of each capability. (This writing is current as of 6.5.2.)

Splunkers With Benefits

As you read these, keep in mind that when you disable these capabilities, users will approach their Splunk experts (hey, that’s you!) for help or even specific access to those features. This becomes a wonderful opportunity for the Splunk expert to collaborate with the user on best practices, thereby improving their Splunk experience and Splunk education. As their skill set builds, they’ll eventually demonstrate enough proficiency for you to promote them to a role with some of these capabilities. In turn, they’ll become the Splunk expert for their team, answering usage questions so you don’t have to. It’s like a pyramid scheme…or viral…or something – the point is, you get to refocus on your work because you’ve empowered your user base.

Accelerations: accelerate_datamodel, accelerate_search, output_file

Acceleration of Reports and Data Models are amazing. They make an otherwise sluggish search return in superman speed…BUT, at a cost of compute and storage. That’s right, when such an object is configured for acceleration, searches run in the background to generate those acceleration details, and then those details are stashed on disk for you. And of course, those searches count against your concurrent search capacity, both by user, and the overall system…until someone disables the acceleration, which in my experiences, is never the user cleaning up stuff they no longer use (as if).

Similarly, disk usage can quickly disappear if users are creating (and never destroying) uniquely named (like using the date stamp) lookup files with every run of a search.

Scheduled Searches: schedule_search, schedule_rtsearch

If you disable the scheduled searches, your users can’t implement alerts. But that also means they can’t cause Splunk to get sluggish running their mal-formed search when they’re not around to appreciate how bad it is. Even more, if you investigate some of the alerts you have in your environment, you may find that many are e-mail alerts that go to folks that aren’t around anymore, or others who’ve set up mail rules to delete the alerts.

Real Time Searches: rtsearch

In reality, a realtime search isn’t necessary. I’ve yet to come across a search that couldn’t instead just be rerun as frequently as needed. This is important to understand given the impact realtime searches have on the environment – a ton! A realtime search is a persistent search process on the Search Head as well as any Indexers. In contrast, a non-realtime search only persists until the search completes and therefore returns resources to the system for other users. By disabling realtime search, you mitigate the [eventual] situation where a group of users all load the same dashboard where every panel is a realtime search, thereby consuming all resources in the Splunk environment and preventing other usage. Plus, as of 6.5.0, any panel in Splunk can be set with a periodic refresh (in the Edit Search pane) to automatically rerun the search for the latest data.

Search Limits: srchJobsQuota, srchMaxTime, srchTimeWin, srchDiskQuota, rtSrchJobsQuota

These may not be necessary if you’ve embraced changes with the other capabilities highlighted here. On the other hand, they can be super helpful to limit the impact of poor Splunk behavior by putting a cap on the quantity of concurrent searches, how long those searches might go on for, how far of a time window of data can be searched, and/or how much storage is used to retain the results of the search.

Be Practical

In the end, see what works best for your organization and your company culture. If you modify these capabilities, remember that initially you’ll do more “teaching” to your users, but only until they learn enough from you and are promoted to a role with more capabilities and features – then, the student becomes the teacher as they empower their peers directly.