Working with AWS Cloudwatch Metrics

The AWS Application provides a very rich set of dashboards to get visibility and an understanding of what is happening inside AWS environments. However, what if you want to see a broad view of all the metrics, or even start visualising metrics of services that the AWS Application doesn’t provide? For example, you may have a set of micro-services running in AWS ECS, some database services running in RDS, and some EC2 instances to monitor. With cloud environments being much more dynamic and elastic than on-premise, having a workspace that is intuitive and flexible to use, and that can bring log events and metrics in the same view, is a great asset to any DevOps team.

Splunk’s Metrics Workspace provides a visual, easy to use interface for quickly discovering, analyzing and acting on metrics datasets. The new release of the AWS Add-on is able to ingest Cloudwatch Metrics directly into Splunk’s Metrics store. Using these two new features allows you to monitor, investigate and easily set up alerts of Cloudwatch metrics, adding to the rich features of the AWS Application.

So how do you set it up?

You’ll need to be running Splunk 7.2 or later, and have installed and set up the latest version of the Splunk Add-on for AWS (version 4.6). The Add-on will require some set up of AWS permissions so that you can connect and collect from AWS’ services (see here for details). For Splunk Cloud customers, you will need a Heavy Forwarder installed in your own VPC with this Add-on installed.

You’re now only three simple steps away from visualising AWS Cloudwatch metrics with Splunk’s metrics workspace app:

Step 1: Install the Metrics Workspace App on your Splunk Instance - here.

Step 2: Create a new metrics index. See ‘Get Started with Metrics’ for more details on this.

Step 3: Navigate to the AWS TA in Splunk, and open the Inputs tab. You will need to add a new input – click on “Create New Input” and select the “Cloudwatch” input:

In the AWS Input Configuration section, populate the Name, AWS Account, Assume Role, and AWS Regions fields, based on your configuration. In my example below, I will be monitoring Cloudwatch Metrics from Ireland and London Regions.

Navigate to the Splunk-related Configuration section, and in the Source Type field, type aws:cloudwatch:metric.

Click on the Index dropdown menu – note that you will need to type in the name of your metrics index (it will not appear automatically in the dropdown). Click Save.

You may wish to change the polling interval to request Cloudwatch Metrics in the Advanced Settings. By default, Splunk will set this as 300 (seconds), but remember that setting this less than the interval times in AWS Cloudwatch will not increase the actual data available from AWS.  

Note also, if one of the metrics you wish to capture from AWS is not in the list (or you don’t want to pull one of the metrics already listed), you can click on the link (Edit in advanced mode) and enter the metrics Namespace in the advanced namespace panel.

Your Cloudwatch Metrics data should now be available in Splunk (you may need to wait a few minutes for the data to initially load).

Note - The AWS Application still expects Cloudwatch Metrics to be in event logs. However, you can consider using the new “mcollect” SPL command to extract metrics from your existing event logs into a metrics store index. This is useful for converting your historical log data into metrics, enabling you to take advantage of the metrics workspace. This blog describes how in a little more detail. You can also use both inputs (metrics and events), but note that you will have the indexing “cost” for both sources.

Test and Start Visualising!

To test this, open up the Search & Reporting app. You should have a “new” tab available called Metrics – this the new Metrics Workspace! Click on this tab.

On the left side you should now see some of the metrics from Cloudwatch available to work with (in my case I have some EBS, EC2, ELB and Lambda metrics available):

Selecting any of these 'dimensions' will open up a timeline chart with the metrics displayed - no search query or SPL is needed!

You can now save all these panels individually, or as a collection onto a single dashboard. Behind each panel is still an SPL search (you can open up as a search to view the SPL). You can also set up alerts, such as the one I’ve set below to trigger if the CPU average is >85%:

Hopefully you’ll have seen how easy it is to now ingest AWS Cloudwatch Metrics into Splunk’s Metrics Store, and how quickly the Metrics Workbench can be used to visualise these. For more information on the metrics store and metrics workspace take a look at these previous blog posts:

Metrics to the Max: Dramatic Performance Improvements for Monitoring and Alerting on Metrics Data

Accelerating Time to Insights with Splunk Metrics Workspace

Thanks for reading!

Paul