How We Built a Splunk App to Help SDC Developers Monitor Incoming Data

This is a guest blog post from Tom Kopchak and Steve McMaster at Hurricane Labs.

Hurricane Labs is a Managed Services Provider that has a meaningful approach to enhancing our customers’ Splunk and security core. We build and publish a number of apps to Splunkbase—notably the Broken Hosts App for Splunk and Cisco Umbrella Add-On for Splunk—as well as provide managed services for enterprises of all sizes and deployment types.

We were excited when Splunk contacted us about the Splunk Developer Cloud (SDC) beta. The first thing that came to mind was building a version of the Broken Hosts App for Splunk with the goal of providing a reliable alert to every SDC developer whenever a host stopped sending data to Splunk.

Everything we do is Splunk, so we always want to make sure we know what’s going on as soon as possible. After hearing about the SDC preview, we realized we could help build an app that everyone on the new platform would need.  

Today, on Splunk Enterprise, the Broken Hosts app runs a giant saved search every 30 minutes that looks at the last time a log was received for each index/sourcetype/host and alerts if that time is later than ‘expected.’ The results of these searches are then visualized in a SimpleXML dashboard, which allows our customers to quickly get a visual representation of the current status of hosts, in addition to an email alert with a list of everything that’s ‘broken.'

At first, we planned to do a straightforward port of the Splunk Enterprise Broken Hosts app to SDC. Unfortunately, that wasn’t possible given the new application building model and several updates to SPL v2.

It was at that point that we took a step back to consider what opportunities the new platform presented in order to do something different.

Once we shifted our mindset, we found several benefits for developing on SDC:

• Simple, easy-to-use REST API: In terms of development, we used the JavaScript SDK for our app front end, but we didn’t use either of the Splunk-provided SDKs for the backend. Instead, we elected to use Python for our app backend and hit the API directly. The API was simple and documented well enough that we were able to implement all the calls ourselves in Python.

• SPL v2: We realized that we could break out the one monolithic search we run on Splunk Enterprise into several separate, more efficient ones. Instead of running a giant saved search on a fixed time interval, we can dynamically schedule searches to match the desired use case. This is because we know from the configuration what data should come in at what interval. It was fairly easy to get started writing queries in SPL v2 because it felt familiar to SPL v1 while also introducing cleaner, more structured syntax.

• New dashboarding framework: The dashboarding framework in SDC has moved to being React-­based without losing any of the key functionality we depended on with SimpleXML on enterprise.

• Fast time to value: Building the original Broken Hosts app on Splunk Enterprise took multiple months, but we were able to develop a new version for SDC in less than three weeks. Minus the planning time, it only took three days for the backend and one week for the front end.

During the remainder of the beta period, we’re planning to build more recurring options, start using the SDC Action Service, explore VictorOps integration, and find ways to apply machine learning to the data patterns for more anticipatory alerting.

If you haven’t applied for the SDC preview yet, you can do that at splunk.com/sdc. And keep in touch with us during the beta period; we’re looking to share the Broken Hosts app for Splunk Developer Cloud with other beta developers and would like to hear more about what you’d like to see!

Steve & Tom ­­­­

P.S. Check out the press release talking about how our team’s Splunk ninja skills helped us bring home the win at the Splunk .conf18 Boss of the NOC competition!