Splunking Microsoft Azure Monitor Data – Part 1 – Azure Setup

Updated 2/22/2021: Splunk support to import Azure data has been updated. Please refer to the information on the Splunk Add-On for Microsoft Cloud Services Splunkbase page for more details.

Azure Monitor is a platform service that provides data routing and access for Azure resources.  Azure Monitor exposes 3 main types of data:

1) Metrics – these are typically performance metrics

2) Diagnostic Logs – logs generated by a resource

3) Activity Logs – who did what and when in the Azure environment

In order to get this data into Splunk, certain setup steps need to happen on both the Azure side and the Splunk side.  This blog post focuses on what needs to happen on the Azure side.  We will look at the Microsoft Azure Add-on for Splunk for data collection in a later blog post.

How Azure Monitor Data is Exposed

Activity logs are generated by the Azure control-plane.  This is basically who did what and when in your Azure environment as well as information like service health, recommendations, security and more.  Azure Monitor routes activity log data to an Event Hub. 

Diagnostic logs are generated by a resource after it is provisioned.  The contents of a diagnostic log will differ depending on what type of resource was provisioned.  As an example, a Key Vault diagnostic log will contain an event when a secret is retrieved via a "Get" operation.  Diagnostic data can be routed to more than one place, but for the purposes of this blog, the data will be routed to an Event Hub.

Metrics are numerical values generated by an Azure resource to help you understand the operation and performance of the resource.  Metrics are accessible via a REST API and are available for 93 days (as of this writing).

For more information about getting additional Azure data sources into Splunk, refer to the Getting Microsoft Azure Data into Splunk blog post.

Azure Components

There are two main components needed on the Azure side:

1) Azure AD Application – I like to think of this as a user or service account with a user ID and password - except an Azure AD application has an application ID and key.

2) Event Hub – this is the central location where activity and diagnostic logs will be stored.

We will be granting the Azure AD application certain roles and permissions during the setup process.  This is done by creating service principal objects.

Going back to the user analogy, after creating a user, you often need to grant them permissions to areas in your network.  To do this for an Azure AD application, you create a service principal object.  The service principal defines the access policy and permissions for the application in a tenant.  So, an Azure AD application is like a user (with an ID and password), and a service principal object is like the user's level of access.  This is an oversimplification of course, so here are the technical details.

Event Hub Security

In order to programmatically pull data from an Event Hub into Splunk, you need an Event Hub connection string and an Event Hub name.  An Azure AD application is not necessary for Event Hub integration.

Metrics Security

Metric data is pulled from Azure via a REST API.  An Azure AD application ID and key are needed to authenticate to the REST API and access the Azure Metric data.  

Setup Overview

Now that you know what Azure components are needed, let’s look at an overview of the setup:

1) Create an Azure AD application registration.

2) Create an Event Hub Namespace.  Azure will write activity logs and diagnostics logs to individual Event Hubs contained with this namespace.  You are not limited to one namespace, but for the purposes of this blog, we will just use one.

3) Set up all necessary Service Principal objects (think of these like the rights a user has on a network).  Since Splunk will be getting data from Azure in the context of the Azure AD application, we need to make sure that application has the necessary rights to the sections of Azure that house the data.

Along the way in this guide, some pieces of information will need to be saved for later – either for the Azure setup or the Splunk setup.  Those pieces of information will be called out in each step with this icon 

Ok, let's set everything up on the Azure side now…

Create an Azure AD Application

1. Select Azure Active Directory -> App registrations -> New application registration

Note: an existing Azure AD application registration may be used if desired.

2. Enter a name and Sign-on URL and click Create. 

Note: the Sign-on URL is not used, so any valid URL is acceptable.

3. Create a Key for the Application Registration

Click Settings -> Keys -> enter a key description -> set the expiration for the key -> click Save.

After clicking the save button, make a copy of the value for the key.  This is the only time the value will be displayed in the Azure portal.  If you lose this value, a new key will need to be generated using the above process.  This key will be stored in a Key Vault later.

 Save the Application ID and Key.

Grant the Azure AD Application Reader Access to the Subscription

After creating the Azure AD application (which is similar to a service account), the application needs to be granted reader access to the subscription(s) via a service principal object.

1. Select Subscriptions -> your subscription -> Access control (IAM) -> Add -> select the Reader role -> type the name of your application registration from the previous step -> select the application when it appears in the results -> click the Save button.

If you have multiple subscriptions that you would like to use with Splunk, repeat the above step for each subscription.

Create an Event Hub Namespace

The Event Hub Namespace will contain one or more Event Hubs.  The configured Azure services will create Event Hubs in this namespace to store activity logs and diagnostics logs.

1. Select Event Hubs -> Add -> supply a Name, Resource group, and any other settings -> Create

2. After the Event Hub Namespace is created, we will need to get the connection string

Select your Event Hub Namespace -> Shared access policies -> RootManageSharedAccessKey -> copy the Connection string–primary key

 Save the policy name and primary key.

Recap

We now have all the plumbing set up, so let's take a moment to recap what has been done so far:

1) Created an Azure AD application registration.  This is similar to a user or service account where the Application ID is like a user ID and the application key is like a password.

2) Granted the Azure AD application reader access to our subscription(s) via a Service Principal object.

3) Created an Event Hub Namespace.

Now, we will expose some data in Azure

Send Activity Logs to an Event Hub

1. Monitor -> Activity log -> Export

2. Select your subscription and regions to export -> set a retention -> check the "Export to an event hub" box -> Service bus namespace -> select the Event Hub Namespace and policy name created earlier -> OK -> Save

Send Diagnostic Logs to an Event Hub

Diagnostic logs are configured at the resource level and can be configured via the Azure Portal, PowerShell, or Azure CLI.  Here are more details on enabling diagnostic log settings.

As an example, we will enable diagnostics logs for the Event Hub we created earlier.

1. Event Hubs -> select your Event Hub -> Diagnostics logs -> Turn on diagnostics

2. Enter a name -> check the Stream to an event hub box -> select your Event Hub and access policy -> OK -> select which logs you would like to send to the Event Hub -> Save

Conclusion

This wraps up everything that needs to happen on the Azure side to make data readily ingestible to Splunk via the Microsoft Azure Add-on for Splunk.  We have set up an Azure AD application, Event Hub Namespace, Service Principals, Activity log settings, and Diagnostics log settings.  In the next blog post, we will look at installing the Microsoft Azure Add-on for Splunk and setting up data inputs to pull down all of the data we made available in Azure.