Splunking Microsoft Azure Monitor Data – Part 2 – Splunk Setup

Updated 2/22/2021: Splunk support to import Azure data has been updated. Please refer to the information on the Splunk Add-On for Microsoft Cloud Services Splunkbase page for more details.

Azure Monitor is a platform service that provides data routing and access for Azure resources. Azure Monitor exposes 3 main types of data:

1. Metrics – these are typically performance metrics
2. Diagnostic Logs – logs generated by a resource
3. Activity Logs – who did what and when in the Azure environment

In order to get this data into Splunk, certain setup steps need to happen on both the Azure side and the Splunk side. My previous blog post, "Splunking Microsoft Azure Monitor Data – Part 1 – Azure Setup," covered the steps necessary on the Azure side for Azure Monitor data collection. This blog post focuses on what needs to happen on the Splunk side.

Setup Overview

Fortunately, the Microsoft Azure Add-on for Splunk has inputs for all three types of data mentioned above (plus a lot more). However, there are a few extra steps involved in setting up this add-on. The following is an overview of what we will be detailing in this blog post:

1. How the Microsoft Azure Add-on for Splunk gets data
2. Set up an input for Event Hubs
3. Set up an input for Metrics

How the Microsoft Azure Add-on for Splunk Gets Data

If you were following along in the previous blog post, you would have ended up with a table of saved information like the following:

Recall that activity logs and diagnostics logs are going to be routed to an Event Hub by Azure Monitor. In order to access the Event Hub data programmatically, you need an Event Hub name and connection string. 

In order to access the Metric data programmatically via the REST API, you need an Azure AD application ID and application key.

Set Up an Input for Activity Logs and Diagnostic Logs

After installing the add-on in your Splunk environment, open the add-on and navigate to Inputs > Create New Input > Azure Event Hub

The following screen is presented prompting for input parameters:

We already have the connection string.  The next piece of information you need is the Event Hub name.  When you create the diagnostic setting on the Azure side, you have the option of specifying an Event Hub name or letting Azure create one for you.  If you let Azure create the Event Hub for you, activity and diagnostic logs are typically written to the insights-activity-logs hub.

Here is a search that shows the creation of the Event Hub namespace from the previous blog post:

sourcetype=azure:eventhub operationName="MICROSOFT.EVENTHUB/NAMESPACES/WRITE"

Set Up an Input for Metrics

1. Open the add-on and navigate to Configuration > Account > Add

2. Fill in the Application ID for the Client ID and the Application Key for the Client Secret.  Then, click the Add button.

3. Navigate to Inputs > Create New Input > Azure Metrics

You will be presented with the following screen:

• Specify a unique name
• Specify an interval (in seconds).  This is how often the input will collect metric data.
• Select the desired index
• Select the account created in the previous step
• To get the Tenant ID, open the Azure Portal: Azure Active Directory -> Properties -> Directory ID

• To get the Subscription ID, open the Azure Portal: Subscriptions -> subscription name -> Subscription ID

• Specify one or more metric namespaces.  For example, to collect Virtual Machine metrics, specify Microsoft.Compute/virtualMachines.  For a complete list of available namespaces, refer to this document.
• Select desired metric statistics (Average, Minimum, Maximum, Total, Count)
• Select your preferred aggregation time grain.  If the preferred time period is not available for a specific metric in the namespace, the next available time grain will be used.
• Specify the number of threads to use for metric collection.  Each individual thread will download metric data in parallel.

Here is a search displaying Virtual Machine metrics:

sourcetype="azure:metrics" namespace="microsoft.compute/virtualmachines"