Enabling debug messages

Splunk spits out an astounding number of its own internal log messages, some I’ve already described. This post is how to get more of them, in case you have spare disk space lying around and need something to fill it with. Or you have some problem with Splunk and need debug logs. Sometimes Support will ask for this to diagnose an issue.

splunkd log messages go in the file splunkd.log. (Note that if you move the existing file out of the way, a fresh one is created on startup if you want to work with only the messages from the current run.) They are controlled by the log.cfg file located in /opt/splunk/etc, which specifies the log level of messages by category:

rootCategory=WARN,A1
category.LicenseManager=INFO
category.TcpOutputProc=INFO
category.TcpInputProc=INFO
category.UDPInputProcessor=INFO

Messages can be set to, in order of severity: DEBUG, INFO, WARN, FATAL, CRIT. Setting a log level gets you messages at that level and higher, so default settings are typically INFO or WARN. When you change something in this file, you need to restart Splunk for it to take effect. When you restart with the –debug flag, it uses a similar file, log-debug.cfg, with a different set of settings for DEBUG messages. Not everything is set to DEBUG, because some of the categories are very chatty.

One of those is FileInputTracker, which even in log-debug.cfg is set to WARN. If you are having problems with data input from files, either indexing multiple times or not indexing at all, set this to DEBUG to get more about what is going on.

Now there is another way to enable and disable messages other than changing the file and restarting. If you want to permanently change settings, or you need to test a script that manages starting and stopping Splunk, you’ll want to use these files. But you can also turn loglevels for categories off and on with a specially constructed search:

| oldsearch !++cmd++::logchange !++param1++::root !++param2++::DEBUG

This is the seach used for 3.3.x, for 3.2 and before remove the “| oldsearch” part. Yes, that is really the pipe, or vertical bar, character there. (And you will get the message “Search Execute failed because Setting root priority” when the search completes.) You can change any category to any loglevel with this, using the category name for the param1 value and the loglevel for param2. “root” is a special keyword for all messages, otherwise use the correct category name like “LicenseManager”. log.cfg is not changed, and on restart you will revert to the configured settings.

One clever thing you can do with this is set up a scheduled saved search to turn on debugging only when you want it. If you have some problem that you know happens around midnight, you can set up one search to turn it on (set it to DEBUG) and off (return it back to WARN or INFO or whatever.)

splunkweb messages are controlled by a different mechanism, the SplunkWeb.tac file. If your problem is specifically with splunkweb, such as debugging LDAP settings in the UI, turn on these additional messages. You do need to restart splunkweb, but this can be done with “splunk restart splunkweb” rather than restarting splunkd along with it on a normal restart.

Change this line:

# set global logging level
appLoggingLevel = logging.INFO

To this:

# set global logging level
appLoggingLevel = logging.DEBUG

The additional messages are output in $SPLUNK_HOME/var/log/splunk/web_service.log file.