Indexing Events Delivered by Multicast

Although the title of this entry says indexing events delivered by multicast, the first thing I need to point out is not to do it. If you are indexing log type events, it is not a good idea to multicast this data to all machines on the LAN just to have one Splunk indexer that is listening for it to index it. Since most of the machines on the LAN won’t be interested in this data, It would be a waste of network resources, not to mention potentially unreliable.

Having said that, there may be cases where indexing events, such as control data, that are delivered via multicast may be useful. For instance, application servers in a cluster often are designed to send out heartbeat packets every N seconds, (typically 10 to 20 seconds) to all other servers in the cluster via multicast. A Splunk indexer could be listening on the same multicast address and port to index this control data as it is being sent around. The Splunk user can then use this for troubleshooting, building timechart graphs for trend analysis, setting up alerts, or simply to see the interval a server was down from the time it sent a packet to the next time it sent a packet.

Setting up a Multicast Listener

As an experienced Splunk user may imagine, the most straight forward way to accomplish this is to create a scripted input that starts when Splunk starts and calls a program to listen on a multicast address and port. Every time the program receives an event, it outputs an entry to standard output, which is then indexed by Splunk. The first step is to set up an inputs.conf file in one of your apps or in $SPLUNK_HOME/etc/system/local that points to your scripted input.

# Windows
[script://.\bin\multicastReceive.bat]
interval = -1
sourcetype = multicast_entry
source = multicast_packet
disabled = true

# Unix
[script://./bin/multicastReceive.sh]
interval = -1
sourcetype = multicast_entry
source = multicast_packet
disabled = false

Notice, the interval is set to -1. This means run this program once upon starting Splunk. The shell script or bat file will call your multicast listener program that will listen on an address and port. Be sure to set the path of the script correctly if you not using a real app and are using $SPLUNK_HOME/etc/system/local to properly point to your script. Here’s my Unix version of the called script, multicastReceive.sh

#!/bin/bash
JAVA_COMMAND="/usr/bin/java"
SPLUNK_HOME="/Applications/splunk3410"
CLASSPATH="$SPLUNK_HOME/etc/apps/multicast/lib"

MULTICAST_ADDR=”237.0.0.1″
MULTICAST_PORT=”8001″

$JAVA_COMMAND -cp $CLASSPATH mcreceive $MULTICAST_ADDR $MULTICAST_PORT

Notice that I’m actually using a publicly available Java program to receive multicast events. This is all you have to do. I’ve modified the Java program’s output to format events in a fashion I would like to use. If I do a search on sourcetype=multicast_entry, my results look like this:

Mon Dec 14 17:16:10 EST 2009 bytes=9 Hi There!
Mon Dec 14 17:16:12 EST 2009 bytes=12 Control Data
...

The complete app add on can be downloaded from Splunk.com’s app add on section. There is also a mcsend program to test sending out multicast events. The authors of this publicly available Java code and their site are acknowledged in the READ ME text file. As always, this is just an example on how to do this and the app add on along with the publicly available Java code comes as is. If you have a need to index multicast data, you can start with the approach outlined here.