Carrot vs Stick: A Case for Incentive Driven User Access

Splunk is committed to using inclusive and unbiased language. This blog post might contain terminology that we no longer use. For more information on our updated terminology and our stance on biased language, please visit our blog post. We appreciate your understanding as we work towards making our community more inclusive for everyone.

Houston, We’ve Got A Problem

Out of the box, a Splunk user has the capabilities to do some powerful stuff – but as Uncle Ben tells us, “with great power comes great responsibility“. In this post, we’ll review the scenario and purpose behind Incentive Driven User Access. In a future post, we’ll dive into the conf files and explore what settings are worth reviewing to implement such a solution.

Scenario: Bort is a well intentioned user at Gift Store, Inc. (an ecommerce known for its novelty stores from the 1990’s). Soon after getting his Splunk access, Bort starts throwing down some awesome real time searches and learning some sick new insights from his data. Bort is finding his searches in Splunk so valuable that he decides to set an alert from one of the real time searches to email him when the eStore runs out of “Bart” novelty license plates. Then, to impress his teammates (who are obviously also all named Bort) he shares a dashboard he built that contains many real time searches showing different sales metrics and system stabilities. Unfortunately for Bort, as soon as his co-workers open the link to view the dashboard, the entire Splunk platform comes to a crawl.

Obviously, Bort is freaking out, and even more, without the real time alert he set up, the eStore might run out of it’s inventory of “Bart” novelty licence plates right when a customer wants one! Fortunately, Bort’s Splunk Admin, John F. (wait, that’s too obvious, let’s go with J. Frink), is an excellent admin and already remedied the situation but at the cost of disabling Bort’s alert and dashboard.

Frink notices Bort’s passion for Splunk and teaches him all sorts of Splunk Fu Ninja Nunchuck Skillz. Thanks to his exploring the data in Splunk, Bort understands the eStore’s business so well that he gets promoted and even speaks at Splunk’s .conf every year. A real dream ending. Just perfect.

But what if it went another way? What if Bort never got a chance to create the real time alert and dashboard? Would Bort have ever come across Frink? Would Frink have ever taught Bort the Splunk Fu he needed to be successful? Even worse, I can’t even imagine a .conf without Bort. That’s just the worst.

But maybe it’s not so bad…

Scooby Doo Ending: We all know Bort. Soon after getting his Splunk access, Bort starts throwing down some awesome historical searches and learning some sick new insights from his data. Bort is finding his searches in Splunk so valuable that he decides to save them. While saving his work, Bort notices that there’s an option to turn his search into an alert – but it’s disabled! Bort decided to reach out to his admin to learn more.

Fortunately, Bort’s Splunk Admin, Frink, is an excellent admin and explains to Bort that while he is happy to convert Bort’s search into an alert, Bort can also gain the capability to do this on his own in the future. Bort is intrigued and learns from Frink that with completion of every Splunk Education course, Frink can grant Bort more capabilities in Splunk.

Bort embraces Frink’s offer and takes it even further, eventually landing his Power User Certification and gaining all sorts of juicy Splunk capabilities. Bort’s proficiencies with Splunk inspires his peers (still also all named Bort) who do the same. Eventually, the entire team is so successful with the insights they gain from Splunk that it becomes a competitive advantage for the company. As a result, the company becomes ridiculously profitable and buys the local Nuclear Power Plant for diversification. Bort’s contributions makes him wildly successful and he retires early to raise horses (although he names each of them Buttercup…for some reason). Don’t worry, in this story Bort also speaks at .conf every year as well.

Oh, and since this was the Scooby Doo Ending, we should point out that Bort eventually found who had been stealing all of the “Bart” novelty license plates. It was Old Man Terwilliger who was dressed up as an intern. Very strange dude. Very strange.

Ok, Enough With The Story

Alright, alright. The point is that instead of reacting to bad behavior, Frink could create incentives that inspire users to be motivated to learn more on their own…and that I really like The Simpsons…but mostly that first thing about incentives.

Out-of-the-box, Splunk empowers users with a ton of access, which most deployments keep as-is. But, as someone who has been the admin of an maxed-0ut Splunk deployment, I can tell you that less is more. By flipping the switch and exploring options available in a Splunk role’s definition, I was able to create a viral solution where interested users earned their way to being power users and further distributed Splunk knowledge to their peers who in turn became Power Users, and so on and so forth.

“Gotchas”

Review the different aspects of a role. Do you want a new user to be limited on how long their search can run for? How far back in history it can search? What data sets they have access to? Ability to consume Splunk resources when they are not directly logged in?

While the documentation elaborates on the ways of controlling those types of things, consider how you can objectively measure a user’s proficiency before promoting their access. You probably don’t want to be in the political battle of subjectively determining when one user appears “qualified” or not. Instead, an objective measure can be used. There’s a ton of options here, but perhaps you require them to show their Splunk Education completion certificate, or a Power User certification, or even a level of Karma points on answers.splunk.com. Any of those individual or together would work – think of it like a game: the user has to earn enough “experience” to level up.

Consider making a Splunk role for data access that is separate from the capabilities. That way you can create various permutations of roles with blends of data access and differing capabilities. This was a brilliant contribution from jnguyen413.

Of course, if you’ve not being doing this already, you’ll want to make sure you have your management’s support so they can back you up when folks complain that they suddenly have less access than they did before you implemented this approach.

Return on Investment

The bottom line is that by embracing such an Incentive Driven Access approach, you can motivate your users to learn more, get more value from the Splunk investment, and run a more efficient Splunk infrastructure (less poor performing searches), which in turn means less hardware needed to run your Splunk deployement.

Thanks!

Thanks again to jnguyen413 other customers who’ve tried this and shared their experiences. Internally, thank you to fellow Splunkers Aly, Tammy, and Dave C. This idea could not have gotten fleshed out without everyone’s contribution.