AWS + Splunk: Ingestion Just Got Easier and More Scalable

We are excited to announce the newest release of the Splunk Add-on for Amazon Web Services, version 4.4.0, and the newest release of the Splunk App for AWS, version 5.1. Before we dive straight into the new benefits and features of these releases, let me provide an overview of what we recently shipped in the previous 4.3.0 release of the add-on since it goes hand in hand. 

Splunk Add-on for Amazon Web Services 4.3.0 Release

The 4.3.0 release includes major enhancements around ingestion speed including a brand new S3 input type. This enhancement removes the need to periodically scan S3 buckets for new inputs, by subscribing to SQS queues for new incoming events notification and optimizing on API consumption. This also presents another advantage around ingestion speed where the add-on detects and ingest events in near real time. 

Moreover, this input type is stateless since checkpoints are no longer stored on the Splunk side, but are now persisted in SQS queues instead. This results in greater fault tolerance and higher scalability where you can now spin up multiple modular inputs and point them to the same SQS queue. If one input goes down, remaining ones can pick up the load. This also means that you can scale out and ingest your data faster by spinning up more inputs as needed. 

In addition to the new S3 input, the 4.3.0 release also includes a new data collection health dashboard that helps you troubleshoot and detect data collection problems. 

How about Splunk Add-on for Amazon Web Services 4.4.0 release?

Now, let's talk about our latest 4.4.0 release which is compatible with the 5.1.0 version of the Splunk App for AWS (more on this below). In this release we made major enhancements around setup usability to make it easier and more intuitive to configure the data collection inputs. Some of the highlights include:

• Better usability and more streamlined configuration workflow
  • The web UI has been optimized, and input configuration UIs for CloudWatch and Config input types let you create multiple inputs in one flow
• Index non-AWS custom logs alongside AWS log types with SQS-based S3 input improvements
  • Assume Role is now supported for all inputs including the addition to SQS, Config Rule, and Inspector input types
• A complete redesign of the Create New Input menu
  • Menu options are now organized by the type of data users want to collect, including enhanced billing input for larger files

Did the Splunk App for AWS get updated too? 

Yes! This is coupled with the release of the 5.1 version of the Splunk App for AWS. In this release, we continued the momentum around cost optimization and made a good thing even better. Some of the highlights include:

• No more input and configuration confusion
  • We removed the input and account configuration from the App, so that now it is all streamlined through the Add-on
• We made it easier for you to plan better to purchase Reserved Instances (RI)
  • Users can now have instance size flexibility in both the RI inventory and RI planner making it even easier to reduce costs by purchasing instances in advance
• Improved visualizations for better dashboarding, analyzing, and alerting
  • Added anomaly custom visualization and enhanced the timeline view for the security group, NACL, Key Pair, Personal Health Dashboards, and more

Looking forward...

Want to find out more?  Please join us at .conf2017 for more in-depth overview of these releases in the session: “Manage Enterprise-Level Amazon Web Services (AWS) Services with Splunk Solution” on Wednesday, September 27, 2017 from 3:30PM- 4:15PM.

Additionally, for more AWS goodness and exciting announcements, please join us at the session: “Gain Real-Time Insights from Your Data Using Splunk and AWS Cloud” on Tuesday, September 26, 2017 from 12:05-PM-12:50PM , where we will be announcing new data ingestion capabilities. To whet your appetite, here’s a quick preview of what we will be covering: serverless, scalability, and enhanced reliability. 

Happy Splunking and looking forward to seeing you at .conf!