What’s New in the Splunk® Dataflow Template

The Splunk Dataflow template is an indispensable tool that allows Google Cloud customers to easily engineer a horizontally scalable and fault-tolerant logging export pipeline into Splunk® Enterprise and Splunk Cloud Platform™. I’m excited to share the news about recent improvements Google has made to this Splunk template across the following areas:

• Better compatibility with the Splunk Add-on for Google Cloud Platform
• Support for Splunk HTTP Event Collector (HEC) fields metadata
• Enhanced Dataflow pipeline reliability, error logging and handling
   

I’m particularly excited about the support for data format compatibility between Dataflow delivered events and the Pub/Sub input in the Splunk Add-on for Google Cloud Platform. For Splunk customers, a common message format means Dataflow sourced events can benefit from the sourcetype assignment and CIM mapping provided as part of the official Add-on.

The Splunk Dataflow template can now also encode the “fields” metadata key in event messages. This is great for customers who want to attach custom indexed metadata to Google Cloud log messages using JavaScript user-defined functions (UDF) prior to Splunk delivery. Imagine being able to populate Google resource labels as Splunk event metadata during log export. Now you can!

Finally, for anyone who has ever struggled to debug UDF failures in a Dataflow pipeline, you’ll be happy to know that the logging and troubleshooting experience has really improved. Instead of silent failures, you’ll find UDF error logs waiting for your inspection in the normal Dataflow worker logs.

While I’ve briefly covered a high-level overview of the Splunk Dataflow template improvements Google has been working on lately, there’s a lot more to learn about.You can read a full explanation of what’s new and improved on the Google Cloud blog. And remember, all improvements are customer-driven, so keep your ideas coming!