Splunk Stream on a Raspberry Pi? YES!

As a network geek, I’ve always wanted to leverage sniffers and deep packet inspection programs to understand user experience and to secure networks. I have a home lab with many virtual machines. But let’s be honest, I really want to know what my household is doing on the Internet! I needed something light-weight, NOT an appliance as large as a data center!

Network Sniffers aren’t anything new. In fact, they’re old school. But, who would have thought a Raspberry Pi would be powerful enough to act as a real-time 24×7 sniffer? I embarked on this journey recently with the Splunk Stream App. And I must say, I’m pretty impressed.

Splunk Stream captures real-time streaming wire data and performs packet analysis at layer 4 (TCP, UDP) as well as many layer 7 applications (HTTP, DNS, etc.) of the OSI model. This enables reporting on things like application response times, application decoding (i.e. what web pages are accessed), as well as detecting unauthorized users.

Splunk Stream is supported on many operating systems, but mostly Intel x86 based computers and architectures. At Splunk, as part of a side project, one of our developers ported the Stream binaries to ARM architecture. PERFECT! The Raspberry Pi is ARM based! On this platform, Splunk Stream is capable of TCP and UDP decoding (layer 4), as well as HTTP decoding (layer 7). Unfortunately, it does not support the breadth of other applications as with Intel x86 versions. However, this was enough for my project. Heck, you can probably install Stream on a rooted android device, as many are ARM based. I don’t know how useful that would be, but it’s quite entertaining! Note that the ARM version of  Splunk Stream is not a GA product, so it’s currently unavailable to customers.

What I used:

Raspberry Pi. (I used the Pi 2 Model B):
https://www.raspberrypi.org/products/raspberry-pi-2-model-b/

USB Wifi Network Adapter (acts the management interface). Here’s the one I got from Amazon:
http://amzn.com/B003MTTJOY

SharkTap:
http://amzn.com/B00DY77HHK

Splunk app for Stream:
https://splunkbase.splunk.com/app/1809/
Part of the purpose of this blog is to showcase how the Stream app is light-weight and can run on a Raspberry Pi! You can install Splunk Stream on a PC or Linux server in lieu of the Pi, and you have the benefit of full functionality, whereas Stream on Pi only summarizes TCP, UDP, and HTTP. The Stream app ported to ARM is currently unavailable to customers, but might be in the future…

Splunk Forwarder for Linux ARM:
https://splunkbase.splunk.com/app/1611/
OR
Splunk Forwarder for most OS’s (to deploy on a PC, Linux, Unix machine):
http://www.splunk.com/en_us/download/universal-forwarder.html

Setting it All Up:

To capture all incoming and outgoing traffic from my network to the Internet, I placed the Sharktap between my service provider (Comcast) and my router (an Asus AC88U). Connected to the mirror port is the Raspberry Pi which has the ‘Splunk Forwarder for ARM processor’ and ‘Stream app ported to ARM’ installed.

Hardware Configuration Diagram:

Collecting Data Diagram:

Command line of the Raspberry Pi with everything installed & running:

The Splunk software installation is just like any other Splunk install. If you need help, check the docs at https://splunkbase.splunk.com/.

The Results:

Now that I have a Splunk forwarder and Splunk Stream running on the Pi, let’s see what shows up in the Splunk interface. From the below screenshots, Stream is seeing TCP traffic, and decoding HTTP traffic. Using the search index=* host=rpi2 sourcetype =stream*, we can see there are stream:tcp and stream:http, as expected.

We also see various others including stream:Splunk_HTTPResponseTime and stream:Splunk_HTTPStatus. These are out of the box sourcetypes that make it easier to track URL response times and HTTP statuses without requiring a lot of Splunk core licensing. They can additionally be aggregated into predefined intervals to further reduce licensing volumes. For example, you can get one record every minute summarizing all response times for a particular URL, or one record summarizing the HTTP status codes by URL. Below is the screenshot for configuring this within the Stream app:

I also have Splunk Stream instrumented on the Splunk server, so it has visibility into all local access, as well as broadcast requests. This is evident in some of the out of box dashboards below:

Why can’t I see security camera video from my phone?

So, now that the Raspberry Pi has been running for a few days and reliably performing deep packet inspection, time to put this data to use and solve some problems. I have a Lorex security camera system on my premise. Through the Lorex Stratus NetHD mobile app, I can see live video streams on my phone and tablet anywhere from the world! However, lately, it hasn’t been working. When registering the camera system in the app, it returns ‘Error Code 43’. This is a perfect use case for Splunk Stream to understand what is communicated between the app, and the security system DVR.

While working from the office (aka Starbucks), I attempt several times to register my security system in the mobile app and from my Apple computer. Next, I access the Splunk interface, and filter on the access ports. I run the following search:

index=* host=rpi2 sourcetype=stream*
(dest_port=xxxx OR src_port=xxxx)
| table src_ip, src_content, dest_ip, dest_content
(ports masked for privacy).

In the table, I can see my 2 source ip addresses hitting the Comcast router. The data sent from the device is encrypted, which is reassuring. But, the contents coming back from the Lorex DVR indicate ‘Page Not Found’. As 2 devices are reporting the same error, I shared these results with Lorex support. The problem turned out to be a recent firmware update with a defect.

I am excited to try other use cases with the Splunk Stream app. I may finally figure out why my Netflix movies sometimes clip. For those of you with kids, you may use this to figure out what your kids are actually doing when they say they’re working on homework. Regardless, post a comment and let us know how you’re using this setup or what you found!

Happy Splunking!

----------------------------------------------------
Thanks!
Ahmed Kira