Integrating with Splunk: You Gotta Think Outside the Box

This morning, a question was asked about integrating with Splunk that started with something like, “but I can’t send syslog from my system, so how can I get that data in Splunk?” It really doesn’t matter what system or what data; before digging in, I already knew that the answer was out there.

“But wait a second, Hal, how could you know that?”, you might be thinking.

Well, it’s just a matter of knowing a bit about how computer systems work, and understanding that Splunk has many ways of ingesting data. You see, at a very high level, there are only two ways that Splunk can integrate with another system. I’ll call these integration types “intentional”, and “operational”. Let’s define them:

1. Intentional – this is when a system causes machine data to be emitted intentionally, for the purpose of exchanging that data with another system. Examples might include:
   1. sending a syslog event
   2. sending an SNMP trap
   3. sending an email
   4. triggering any action, such as invoking a script
   5. APIs, SDKs, scripting toolkits
2. Operational – this is when machine data is created, but not necessarily emitted, simply due to the normal operations of a system. This one is key to understand, because these may be the important sources of data that you didn’t know you even had. Examples:
   1. log files
   2. databases
   3. message queues or busses

Nine times out of ten*, people start with syslog events and end with log files. But there is so much more out there! Let’s say that you have some piece of software which can send emails when something important happens. You like that software, it’s not going anywhere, but you really need to get that important event into Splunk. You’ve already looked at syslog and log files, and don’t know where to go next? Not a problem, we got you! Here are some ideas that might help out:

• Send those email alerts to a mailbox and have Splunk index each message as an event
• Use the Splunk App for Stream to decode the SMTP protocol as email alerts travel over the wire!
• Are the alerts in your native system recorded to a database table? Watch the table with DB Connect!

I don’t mean to bury the lead, but I wanted this post to be general in nature. For those curious, the system in question was Cisco Prime Infrastructure. I’ve never used it, but was able to determine that the above techniques had a good chance of working after skimming the admin guide.

Happy Splunking!

(* I totally made up this statistic.)

----------------------------------------------------
Thanks!
Hal Rottenberg