Splunk Stream 7.2 - Integration with Amazon VPC Traffic Mirroring

Recently, our good friends at Amazon Web Services (AWS) launched an awesome new product, VPC Traffic Mirroring. Here at Splunk, we are excited about this new capability as it allows our Splunk Stream platform to ingest this data, and send it on to any Splunk instance, in the cloud or on premises. Leveraging this capability allows Splunk users to collect specific network data from their AWS environment, and use it to fulfill security, IT Ops, or business-focused use cases.

As Jeff Barr, Chief Evangelist at AWS said:

“You can think of VPC Traffic Mirroring as a ‘virtual fiber tap’ that gives you direct access to the network packets flowing through your VPC…. You can choose to capture all traffic, or you can use filters to capture the packets that are of particular interest to you, with an option to limit the number of bytes captured per packet. You can use VPC Traffic Mirroring in a multi-account AWS environment, capturing traffic from VPCs spread across many AWS accounts and then routing it to a central VPC for inspection.”

Getting started is easy, with just a few configuration items to set up. You’ll need a Traffic Mirror Source, Traffic Mirror Target, Traffic Mirror Filter, and a Traffic Mirror Session. The best way to capture this data is to deploy a Splunk Universal Forwarder with the Splunk Stream TA, or a Splunk Stream Independent Stream Forwarder. The Splunk Enterprise AMI, available on the AWS marketplace, will give you a fast track to achieving this.

An added benefit of leveraging Splunk Stream as the destination for this mirrored data is the built-in filtering and selective indexing capabilities. While AWS offers Target Mirror Filters, these limit the protocols, port ranges or IP address ranges that you can send to the mirror destination. Splunk Stream enhances this capability by allowing a user to specify the specific fields within a protocol stream that are to be ingested, therefore restricting data capture to only data deemed usable and valuable to an organisation. Take HTTP as an example — the default Stream configuration offers users 100+ fields to capture and ingest into their Splunk environment. After review, a user might decide they only need a subset of these fields. They can select just the data needed, and push these out to the configuration. This filtering, or ‘selective capture,’ will be performed at the source (or in our case, the Target Mirror), taking the load off the indexing tier.

We can also apply a filter set via the Splunk Stream UI to look for particular strings within a field itself. For example, users can filter out all web requests that contain the string splunk.com in them. Perhaps, the user is not not interested in seeing these events for her given use case, or perhaps these are the only ones she’s interested in seeing.

Web traffic offers Splunk users a rich data source, with organisations around the world building use cases for all functional units within the business, and applying a different lens to the same data each time. Splunk Stream offers a uniform, scalable method to capture this data, along with dozens of other protocols. The Splunk Essentials for Wire Data provides examples across several different categories, showcasing the use of data captured by the Splunk Stream platform.

The data presented to a Splunk Stream capture interface via VPC port mirroring could be used for many use cases.

For example, IT Operations analysts can use this data for visibility, troubleshooting and reporting of:

• Calls to Amazon S3 data stores (latency, thruput, errors)
• Database query concurrency
• Total email volume
• Application usage reporting (Splunk Stream currently contains signatures for over 300 commercial applications)
  
  

One of our favorite data sources here at Splunk — and applicable across multiple use cases — is DNS data. AWS has made capturing this data across a VPC simple. However, there is a small catch required to ensure this data is sent via a VPC port mirroring session. When setting up the traffic mirror filter to apply to the mirror session, click the ‘modify network services’ button in the console:

In this menu, enable the ‘amazon-dns’ service, which is used to service DNS requests within the AWS environment.

Now that we are capturing DNS data, we can begin exploring specific use cases. Several examples of this are called out in the Splunk Essentials for Wire Data app, available on Splunkbase. These include detecting domain spoofing, and detecting indicators of compromise via entropy analysis.

Thanks for reading, and happy Splunking!