New Features for Perfmon in Splunk 6

Continuing our delve into the new Microsoft features that were introduced in Splunk Enterprise 6, let’s take a look at performance metrics. We had a couple of really cool features here, dealing with zero values, object recognition, and multi-kv.

The first has to do with how we record zero values. By default, the perfmon data input does not record zero values. This has implications if you want to do things like 95th percentile as I discussed in this blog post. In that blog post, I suggested you might want to (shudder) alter the splunk-perfmon.path file to provide a command line argument. The developers obviously read my blog and decided to make it easier. Now you can just add a parameter to your inputs.conf stanza, like this:

[perfmon://MemoryStats]
object = Memory
counters = *
instances = *
interval = 60
showZeroValue = 1

The important bit is that last parameter. If it is set to 1, then the zeros are recorded for each counter/instance combination. The default is the old behavior (i.e. showZeroValue = 0).

Our second feature is the ability to include regular expressions in the object. Let’s take an example of Microsoft SQL Server. If you create a named instance, the SQL Server will maintain counters that look like this:

MSSQL$INSTANCE:Transactions

The INSTANCE is the instance name. This presents a problem when you are trying to provide a generic add-on that reads performance data for Microsoft SQL Server. Our developers came to the rescur again – we can now specify a regular expression. If the object name matches exactly then the regular expression is never consulted. However, if there is no exact match, the object turns into a regular expression match and all objects matching the regular expression are added to the monitoring list. You can do things like this:

[perfmon://MSSQL:Transactions]
object = MSSQL[^:]*:Transactions
counters = *
instances = *
interval = 60
showZeroValue = 1

Our final update to perfmon collection deals with an output change. Normally, you will get events that look like this:

10/15/2013 13:39:04.044 -0700
collection=Processor
object=Processor
counter="% Processor Time"
instance=_Total
Value=2.2639598290484009

This is good, but what if you have a lot of perfmon to gather – thousands of events? This is not exactly the most efficient method of storing the data. Let’s take a look at an example for an alternate mechanism:

[perfmon://Memory]
object = Memory
counters = *
interval = 60
mode = multikv
showZeroValue = 1

That new parameter – mode – switches between multikv and simple outputs. A multikv entry looks like this:

0	33.85326809566461	7734845440	7975034880	14696304640	0	0	3.982737423019366	29.870530672645245	1.991368711509683	1.991368711509683	1.991368711509683	0	299855872	94121984	0	312774	180313	33556298	66895872	89628672	290975744	0	9474048	10956800	8876032	66895872	54.26557951373551	7553560	7376	0	1615249408	1789952	31371264	6088224768	0	14400

Ok – it isn’t the most readable event in the world. But try this:

sourcetype=PerfmonMk:Memory| table Available_MBytes,Committed_Bytes,"Demand_Zero_Faults/sec"

All of a sudden, the information in this table is decoded for you. The format is much more compressed and that makes it take up less license room. Note that the source type has changed (it’s now prefixed with PerfmonMk instead of Perfmon), but other than that – it’s good. What’s more, if you do this to something with instances (for example, LogicalDisk), you still get one event per instance and the first element is the instance name:

C:	82.221877626972017	100744	0	0.893482164153651	0.0089348216415365105	0	0	0.893482164153651	0.0089348216415365105	0.00092000519619113603	0	0.00092000519619113603	9.8839878234777032	0	9.8839878234777032	64775.70259994347	0	64775.70259994347	6553.6000000000004	0	6553.6000000000004	97.408979418055637	1.9767975646955405

All of this makes it very possible to get more performance data to correlate against your other event data and store it in the most efficient way possible. Of course, you do need to upgrade your Universal Forwarder to 6.0, so be aware of that small wrinkle.