Getting data from your REST APIs into Splunk

Overview

More and more products,services and platforms these days are exposing their data and functionality via RESTful APIs.

REST really has emerged over previous architectural approaches as the defacto standard for building and exposing web APIs to enable third partys to hook into your data and functionality. It is simple , lightweight , platform independent,language interoperable and re-uses HTTP constructs. All good gravy. And of course , Splunk has it’s own REST API also.

The Data Potential

I see a world of data out there available via REST that can be brought into Splunk, correlated and enriched against your existing data, or used for entirely new uses cases that you might conceive of once you see what is available and where your data might take you.

What type of data is available ? Well here is a very brief list that came to mind as I typed :

• Twitter
• Foursquare
• LinkedIn
• Facebook
• Fitbit
• Amazon
• Yahoo
• Reddit
• YouTube
• Flickr
• Wikipedia
• GNIP
• Box
• Okta
• Datasift
• Google APIs
• Weather Services
• Seismic monitoring
• Publicly available socio-economic data
• Traffic data
• Stock monitoring
• Security service providers
• Proprietary systems and platforms
• Other “data related” software products

The REST “dataverse” is vast , but I think you get the point.

Getting the Data

I am most interested in the “getting data in” part of the Splunk equation. As our esteemed Ninja once said , “Data First , Sexy Next”.

And I want to make it as easy, simple and intuitive as possible to allow you to hook Splunk into your REST endpoints, get that data , and starting writing searches.

Therefore building a generic Splunk Modular Input for polling data from any REST API is the perfect solution. One input to rule them all so to speak.

Building the REST Modular Input

From a development point of view it is actually quite a simple proposition for some pretty cool results.

For RESTful API’s we only need to be concerned about RESTful HTTP GET requests , this is the HTTP method that we will use for getting the data.

And by building the Modular Input in Python , I can take advantage of the Python Requests library , which simplifys most of the HTTP REST plumbing for me.

Using my Python Modular Inputs utility on Github , I can also rapidly build the Modular Input implementation.

You can check out the REST Modular Input implementation on Github

Using the REST Modular Input

Or if you want get straight into Splunking some REST data , make your way over to Splunkbase and download the latest release.

Installation is as simple as untarring the release to SPLUNK_HOME/etc/apps and restarting Splunk.

Configuration is via navigating to Manager->Data Inputs->REST

And then clicking on  “New” to create a new REST Input. As you can see below , I have already created several that I used for testing.

Configuring your new REST input is simply a matter of filling in the fields

Then search your data ! Many RESTful responses are in JSON format , which is very convenient for Splunk’s auto field extraction.

Key Features

• Perform HTTP(s) GET requests to REST endpoints and output the responses to Splunk
• Multiple authentication mechanisms
• Add custom HTTP(s) Header properties
• Add custom URL arguments
• HTTP(s) Streaming Requests
• HTTP(s) Proxy support
• Response regex patterns to filter out responses
• Configurable polling interval
• Configurable timeouts
• Configurable indexing of error codes

Authentication

The following authentication mechanisms are supported:

• None
• HTTP Basic
• HTTP Digest
• OAuth1
• OAuth2 (with auto refresh of the access token)
• Custom

Custom Authentication Handlers

You can provide your own custom Authentication Handler. This is a Python class that you should add to the
rest_ta/bin/authhandlers.py module.

You can then declare this class name and any parameters in the REST Input setup page.

Custom Response Handlers

You can provide your own custom Response Handler. This is a Python class that you should add to the
rest_ta/bin/responsehandlers.py module.

You can then declare this class name and any parameters in the REST Input setup page.

----------------------------------------------------
Thanks!
Damien Dallimore