When I was an admin, sometimes I wanted to Splunk things, but not in my production environment. Maybe I wanted to add data and define the corresponding sourcetype. Maybe I wanted to mess with some backend conf files. Maybe I wanted to muck around with a new version of a search or dashboard. Whatever the reason, I learned a few approaches that may be obvious for the Splunk Ninjas out there, but not so much for our adorable n00bs. Either way, if you find yourself hesitating to try something Splunky, then this post is for you.

Build a Splunk Sandbox

Ideally, you’re installing Splunk on your local workstation (desktop/laptop), but if your company hasn’t given you access rights to install Splunk, then see if there’s a cloud or virtual machine you can get your hands on. Something internal to your organization is ideal so you don’t have to worry about your data leaving the company firewalls. Other approaches include VirtualBox or Docker (for which you’ll find many blog posts -> http://blogs.splunk.com/?s=docker). They key is, you want something that is going to be easy to access, and easy to rebuild when you break it (yes, “when”…if you’re not breaking it, you’re not taking enough risks). Of course, to install just follow your handy Installation Manual which will even walk you through the download.

Once Splunk Enterprise is installed you’ll want to convert it to the free license so you can use it perpetually without any hassle. Don’t worry about pointing it to any indexers or anything like that. You can certainly configure that later using the Distributed Search Manual, but remember that the more you configure your environment, the more work it is to rebuild when you break it, and therefore the less likely you are to take risks.

Play in your Splunk Sandbox

Looking for sandbox data? Check out the free example data available in the Search Tutorial. In fact, if you haven’t taken the tutorial yet, I can’t stress enough how great it is, so stop reading this and take the free Search Tutorial. I’ll wait…..back? Oh, welcome back? Your hair looks nice, did you get it done? I noticed ;).

Ok, back on topic. What if you want to play with data you see in your production instance of Splunk? That’s easy too! Simply export the results of a search you’ve run. There are some gotchas to keep in mind.

• If the search returned a huge amount of events, then it might take a long time to export and later add to your sandbox. Therefore try reducing the Number of Results to make things more manageable.
• If you want raw events, then select that option when exporting. Raw events are good if you’re trying to define new elements of the sourcetype in your sandbox before putting it in production. Conversely, if you export with any of the other structures that export formats to, then you can play with your data without having to worry about the sourcetype’s field extractions. Using a text editor, take a peek at the results of the different options to see what I mean.

Once you’ve exported the data, add it to your local instance! Now you’re cooking…err, I mean Splunking!

When to Avoid the Sandbox

There are situations in which you won’t need a sandbox but you can still play around without much impact to others.

For example, if you’re creating a cool new search, it might be burdensome to build in your sandbox. In that case, embrace your time selector and the head command. Whenever I work on a new search, I change my time selection to something very small so I know my searches won’t bog down the system. Depending on your data, Last 15 minutes is a good choice but check out the Search Manual’s section on this topic to gain more control over your search window. On the other hand, sometimes the data is not evenly distributed and so the last 15 minutes may have sparse results while on the other hand a few hours ago was too voluminous. Since a wider search window could slam the Splunk system, check out the head command. The head command is awesome because you can do a search over a long time period, but trust that only a specified number of results will be returned. In other words, by using head, I can search over something as extreme as All Time but feel safe that as soon as, say, ten events are returned, my search will terminate and no longer impact the system.

Another scenario where a Sandbox may not be a good fit is if you want to tweak a knowledge object such as a saved search or a dashboard. In those circumstances, fear not! There’s a great way for you to mess with a copy of the object without impacting the original. It’s a feature called Cloning. Once you make a clone of an object, your clone will appear private to your account. That means no one else knows it even exists! Using your cloned copy, you can immediately edit and play with the object knowing that you’re not impacting the system and no one will mistake your copy for their own. If you want to share your clone with your peers, check out the Manual on Managing Knowledge Object Permissions OR, if you have permission from the author/owner of the original object, you can replace the original with your cloned copy and then delete your cloned copy.

Apply Config without Restart

Sometimes even in our sandbox, we get tired of restarting Splunk. Fortunately, there’s a couple of interesting tricks you can do to get your configuration reloaded without said restart. Be aware that some of these aren’t necessarily formal or supported means to reload Splunk and therefore if something looks fishy your best bet is to revert back to the trusty ol’ splunk restart.

• If a sourcetype’s props or transforms have been edited, try refreshing with the extract command’s reload parameter.
• If you’re playing with conf files, often you can reload the configuration using the /debug/refresh URI. Check out the Admin Manual for more details. Be careful, this trick might force SplunkWeb users to have to re-login.
• If its static files on the backend that you’re playing with, then see Advanced Dev Manual for how the _bump URI can save you some restarts.

Hesitate No More!

As you can see, there’s a number of different ways for you to get yo’ Splunk knowledge on. The last tip I can give is that if you find yourself avoiding something in Splunk, take a step back and think about what’s intimidating you. You may find a solution is as simple as a sandbox.

I hope this helped. Good luck, have fun, and happy Splunking!