How To Set Up Monitoring for Your Hybrid Environment

The modern IT landscape consists of many distributed systems, which can pose a challenge if you are responsible for the end-to-end performance of these systems. As a platform engineer today, that is exactly what the job requires. You must juggle between dozens of tools to meet SLAs. This is why a modern solution is needed to bridge the gap between disjointed infrastructure and application stacks…and this is why the Splunk Observability platform was born. It can ingest disparate data sources and provide visibility to critical services that have on-premises and cloud components through a single pane of glass. 

Let’s imagine that you are a platform engineer who needs to provide visibility to a hybrid environment with AWS, Kubernetes, and on-premises components and that you’re already using Splunk Cloud Platform for logs. In this blog, I’ll take you through how you can start monitoring your AWS and Kubernetes environments in Splunk Observability Cloud and seamlessly integrate logs from Splunk Cloud.

Comprehensive AWS Visibility Out of the Box

Connecting to AWS allows you to analyze your AWS data in Splunk Observability Cloud. You can easily integrate Splunk with your cloud provider to easily ingest telemetry metadata. You can also collect Amazon CloudWatch Metrics data and store it in Splunk Observability Cloud, then sync all CloudWatch Metrics data for all services and all regions in use in a given AWS account.

To get started, you’d need administrator privileges in Splunk Observability Cloud and your AWS accounts. Splunk provides a guided setup wizard to help onboard any type of data. It includes step-by-step instructions and links to Amazon CloudFormation templates that create the necessary AWS IAM roles for the AWS integration. You’d simply:

• Log in to Splunk Observability Cloud and select Data Management from the navigation menu.
• Select Add Integration and then the Amazon Web Services tile, which will open the guided setup wizard. 

• Follow three easy steps to establish a connection, including: 
  • Define your AWS connection
  • Prepare your AWS account
  • Establish the connection

Guided setup is also available for Azure and GCP cloud integrations. 

Once the connection is successful, you will be able to see metrics from your AWS services populate on the out-of-the-box navigator and dashboards in Splunk Infrastructure Monitoring.

All your engineering teams will have access to valuable insights about your AWS services, such as an overview of Lambda functions or a heat map of errors in your EC2 instances.

Kubernetes Visibility at Cloud Speed

Splunk Observability Cloud also includes a guided setup wizard for Kubernetes integrations so you can get near real-time visibility to your Kubernetes environment. Similarly, you would select Kubernetes from the Integrations page to get started with the guided setup.

The wizard will walk you through:

• Install Configuration where you select the desired values for Environment, Cluster name, Provider/Distribution, Log Collection & Auto-Instrumentation/Profiling. 
  • If the log collection is set to Splunk Cloud or Splunk Enterprise, you’d need to specify HEC URL and HEC Endpoint. 
• Installation Instructions. Here you’ll be provided steps on how to deploy the Splunk OpenTelemetry Collector Kubernetes Helm chart with your desired configuration values. The Collector is deployed using a Helm chart for the Splunk Distribution of OpenTelemetry Collector for Kubernetes. You will need Helm3, administrator access to your Kubernetes cluster, and familiarity with your Kubernetes configuration to deploy. To send data to splunkObservability, you’ll also need:
  • splunkObservability.accessToken. Your Splunk Observability org access token. 
  • splunkObservability.realm. The Splunk realm to send telemetry data to. The default is us1. 

Once all the steps have been completed successfully, Kubernetes data flowing into Splunk will be visible in the OOTB Kubernetes Navigator and dashboards on the Infrastructure Monitoring page. 

Your engineering teams will gain detailed visibility into your Kubernetes clusters down to node details. Selecting a specific Node for this view navigates to an OOTB dashboard with details on the health of the server. 

Integrating Logs for One Unified Experience

Visibility into your environment is incomplete without logs integration. Luckily, Splunk has you covered with Log Observer Connect. Splunk Log Observer Connect allows engineering teams to query logs in Splunk Enterprise or Splunk Cloud Platform within Splunk Observability Cloud so they can troubleshoot application and infrastructure behavior using high-context logs in the same view. Knowledge of SPL is not required, so users can perform codeless queries on Splunk Enterprise or Splunk Cloud Platform logs to detect the source of problems in their systems, and then jump to Related Content, such as APM services or infrastructure dashboards throughout Splunk Observability Cloud, in one click. Your DevOps teams can focus on logs that are important to them without having to sort through security, network, and miscellaneous logs. 

You can set up Log Observer Connect for Splunk Cloud or Splunk Enterprise. The configuration will require steps to be performed in both Splunk Observability Cloud and Splunk Cloud or Splunk Enterprise, and you’ll need administrator privileges to complete the setup. For instance, to set up Log Observer Connect for Splunk Cloud, you’d start in Observability Cloud:

1. Go to Settings > Log Observer Connect and select Add new connection. 
2. Select Splunk Cloud Platform.

Setup would continue in Splunk Cloud. To configure the Splunk Cloud service account user, you must also have the sc_admin role in Splunk Cloud Platform. You’d follow the instructions in the guided setup to configure a role in Splunk Cloud Platform for the Log Observer Connect service account. 

Once the role configuration is complete, you’d go back to Observability Cloud to complete the Log Observer Connect guided setup. After a successful integration, you’d be able to see logs populate on the Log Observer page within Splunk Observability Cloud.

You’d also be able to query by different filters without the use of any special query language. 

As a platform engineer, you can use all this telemetry data to gain insights into the performance of your hybrid applications. Unleash the potential of your data, and elevate your monitoring and troubleshooting capabilities to resolve issues quickly and provide greater SLAs. Don’t hesitate to contact Splunk if you have any questions or would like to test this out with a free trial.