Tips and Tricks for the noob

Before we dive into the meat of things, let me first explain how this came up…   A newer user/admin of Splunk was attending our conference (#datajourney) and found that there was this convenient command that checked configs.   This command is called btool and it has been in the product for some time.  Now, to me this is old stuff but to him this is very very useful as a newer Splunk admin/user.  It may also build your street cred with the long time users. Well, to his request, I’ll share some of the these tips in this post…  Some of these are CLI commands (noted by ./splunk) and the rest are searches. Here we go…

Checking what Splunk thinks of the config files:
./splunk cmd btool list
./splunk cmd btool list --debug
The above command takes a config file parameter and shows you the ‘implied’ settings.  With the –debug flag, it tells you which location it read them from.
./splunk cmd btool inputs list --debug
The above usage takes inputs (or any base name of a config file) as the parameter.

Checking what is currently configured in Splunk – Use the Rest command in the search query language:
| rest <endpoint>
The rest command will return a description of almost anything that is configured, from jobs to input settings.  The below example shows the running searches:
| rest /services/search/jobs
To find all of the endpoints, you could do some navigation via queries OR directly log in to the REST interface:  https://<your_splunk_server>:8089/ .  Note you will use the mgmt/splunkd port + SSL.   Also, you will be prompted to login.

Change the ports on your server:
./splunk set web-port <port_number>
./splunk set splunkd-port <port_number>
Show what port is being used:
./splunk show web-port
./splunk show splunkd-port
Get a high level summary of your indexes (must be run on the indexer):
| dbinspect
Run the above search command and you will get a nice bucket breakdown and other useful details about your index.

Find out what hosts (or sources or sourcetypes) have sent data to Splunk:
| metadata type=hosts
The above search command will give you the name of the hosts that have sent data to Splunk, as well as the time it received data for the first, last, and most recent event.  This is how you can track if a forwarder is sending recent data.   To make this command make sense, use “| convert ctime(*Time)” to make the epoch time readable:
| metadata type=hosts | convert ctime(*Time)
Run a splunk command remotely:
./splunk <command> -uri https://<remote_host>:8089
Using the -uri option will let you run a splunk command remotely.   This means I can have Splunk running on my laptop and can directly run a command on another Splunk instance by specifying it’s location (with -uri). You will need to specify the hostname and splunkd port, as well as appropriate credentials.

Show what files Splunk is monitoring:
./splunk list monitor
The above command will show you all files that Splunk has configured for monitoring.   Note, that if you  have conflicting “disabled” settings for an input this may show confusing data.  Using the REST endpoint to track the actual file detail is a better way to see exactly what is going on with an input.  However, the above gives good info if you are configuring basic input settings and need to check whitelists/blacklists.

Show me what apps are enabled:
./splunk display app

Well that is it for now, I’m sure I can post a follow up for other tricks.

Happy Splunking!

----------------------------------------------------
Thanks!
Simeon Yep