We're Making Observability Available in Splunk Enterprise!

For you, one or more of these statements (and / or challenges) likely apply to you, and the organization for which you work. Which of these are you hearing or saying?

• We are moving to Observability.
• We are modernizing IT Operations.
• We are moving from Monolith to Microservices.
• We are accelerating our Digital Transformation.
   

Splunk can help you with these in many ways. Today, I am highlighting one way to address many of these statements, specifically with the Content Pack for Splunk Observability Cloud.

We are bringing Splunk Synthetic Monitoring, Infrastructure Monitoring, and Application Performance Monitoring all together for you, and into a single view within Splunk IT Essential Work (ITE Work) and IT Service Intelligence (ITSI) giving you the ability to drill into results with a few clicks, and a deep link into Splunk Observability Cloud in context. 

This content pack provides: 1 Service Analyzer, 25+ Services, 70+ Key Performance Indicators (KPIs), 15+ Dashboards, 5 Glass Tables and 2 Technology Add-On (TAs). As a result, you’ll get results within minutes not days and be able to provide views into your observability data that IT Operations, DevOps, Business & IT Executives can understand and act on. 

It’s FREE with both ITE Work and ITSI, but you might recall that from my blogs on our Content Packs for Microsoft Exchange and Microsoft 365 and 3rd Party APM.

^{Figure 1-1. Observability Content Pack: Executive Glass Table}

How to Deep Link into Splunk Observability Cloud from ITSI in Context

One of the big benefits of the Content Pack for Splunk Observability Cloud, is the ability to deep link into Splunk Observability Cloud from ITSI within 5 seconds in full-context. 

To keep this simple, I’ll illustrate how to do this in 4 easy steps with supporting screenshots along the way. 

Step 1:

With just 2 clicks in the Service Analyzer, you can get to the specific Entity you are interested in.  For this example we are drilling down into "checkoutservice" within APM and looking at the duration metrics as you can see from the below screenshot.

Step 2:
Upon clicking on the desired Entity, you will be brought to the Entity Detail view, which shows the results for that Entity in the Splunk APM Overview and specifically the RED Metrics.

Step 3:
Next, click on the “Open entity information panel” icon.  The additional detail flys-in from the right, providing a lot of information and context, including the deep links...which we will click on the “Splunk APM Tag Spotlight - Errors” link.

Step 4:
Lastly, this brings us into context to the Splunk Observability Cloud, in the “Splunk APM Tag Spotlight - Errors” and showing “checkoutservice” details.

Glass Tables in the Content Pack for Splunk Observability Cloud

Glass tables enable you to visualize and monitor the relationships and dependencies across your IT and business services. You can use glass tables to create dynamic contextual views of your IT topology or business processes and monitor them in real-time. You can add metrics like KPIs, ad hoc searches, and service health scores that update in real-time against a background that you design. Glass tables show real-time data generated by KPIs and services.

The Content Pack for Splunk Observability Cloud includes five (5) preconfigured Glass Tables you can use to gain insights across your key Services, KPI’s, and Entities. There are a number of personas which this is relevant, some we have highlighted below, along with why this is important to them.

1. IT Operations: As a member (or Head) of the IT Operations team, I need to have visibility into the performance and availability of our applications and infrastructure to quickly repair or report issues to be remediated, so that I can ensure services are delivered to all customers without interruption.

2. CIO / CTO: As the CIO and CTO(s) or member of these offices, I need to ensure full visibility and awareness of our key systems and applications, ensuring our business leaders are able to operate our business without interruption or incident.

3. Business Leader: As a Business Leader or member of the business team, I need to know End User Experience and Application and Infrastructure Performance, so we can operate our business effectively and serve our customers with speed and efficiency.

4. Support Analysts: As a member (or Head) of Customer or Operations support team, I would like immediate insights into the status of all key services across our business, enabling our group to best serve our constituents quickly and with accurate information.

5. DevOps / SRE: As a member (or Head) of the Engineering team, I need to ensure services are delivered to all customers without interruption, providing my stakeholders with visibility into the Performance and Availability of our applications and infrastructure will allow for them to be best informed and aware.

6. DevSecOps: As a member (or Head) of the DevSecOps team or practice, I need to provide insights into the lower and production environments, to increase accountability for security to deliver secure services faster with reduced business and customer risk.

Within the Content Pack for Splunk Observability Cloud, we have provided the 5 Glass Tables: [please do remember it is simple to clone and make these examples your own, often only taking minutes]

1.  Executive Glass Table

2.  Overview [Columns]

3.  Overview [Layered with Radio Gauges]

4.  NOC Glass Table [Layered]

5.  DevOps SRE Detailed View

Glass Table: Executive Glass Table

The Executive Glass Table, delivers rolled-up insights across 3 major observability areas: Synthetic Monitoring, Infrastructure Monitoring, and Application Performance Monitoring. 

In a single view, you get a quick and real-time update as to what is going on across all of your landscape, providing single-click access to additional details.

This answers the question of what is going on, why it’s happening and provides a deep-link (in context) to the source system in the Splunk Observability Cloud. This dramatically improves MTTI / MTTD / MTTR by simplifying the process of getting to the ‘needle in the haystack’ and resolving the root cause issue quickly.

In the bottom left corner, you will also find key trends across: Total, Synthetic, Infrastructure Monitoring, and Application Performance Monitoring. This makes it easy to understand the history across your applications and infrastructure and customer experiences with a simple glance.

How do you get these insights today?

^{Figure 1-2. Observability: Executive Glass Table}

Glass Table: Overview [Columns]

This Glass Table provides a quick overview of the 3 major elements of: Synthetic Monitoring, Application Performance Monitoring, and Infrastructure Monitoring.

Next level down insights with a single-click to the details delivers immediate insights into a summary across all stacks, across all environments. 

After you click into the details, with another click you will move into the Splunk Observability Cloud with an embedded deep link, providing you with the native results fully in context. Getting you to root cause in just 2 clicks and seconds.

^{Figure 1-3. Observability: Overview [Columns]}

Glass Table: Overview [Layered with Radio Gauges]

This Glass Table provides immediate insight and results, which can be viewed at a distance, with the ‘Radio Gauges’ on the left side making it simple to know where Synthetic Monitoring, Application Performance Monitoring, and Infrastructure Monitoring results are for your entire environment.

These types of insights deliver immediate value to a variety of teams and stakeholders. Click on any of these KPI’s, and get to the details. Click again and get to the Splunk Observability Cloud native results fully in context.

By looking at this Glass Table, I would quickly click to understand what is going on from my Synthetic Monitoring results, but I can also see there is something going on with AWS in my Infrastructure Monitoring Health Summary...which is likely the root cause, and I know to perhaps click there first.

Is this something you can do today?

^{Figure 1-4. Observability: Overview [Layered with Radio Gauges]}

Glass Table: NOC Glass Table [Layered]

The modern NOC (Network Operations Center) is evolving to Observability and increased awareness of what is going on across all stacks within their Users, Infrastructure, Critical Business Flows, and Applications.

An interactive Glass Table enables them to get these results real-time, and proactively manage situations / episodes as they occur, being able to quickly find and resolve issues.

This simplified view is designed for a 40-foot wall or hallway monitor, enabling visibility and awareness, showing the trend and history along with current state for all your critical KPI’s rolled up.

^{Figure 1-5. Observability: NOC Glass Table [Layered]}

Glass Table: DevOps SRE Detailed View

For sure my favorite Glass Table to create. The value of this to organizations is huge!

Here you gain insights across Splunk Synthetics, Application Performance Management, and Infrastructure Monitoring; not only at a summary level, but also two levels down, and ‘single click’ access to additional details for each. 

Within the Browser of Synthetic Monitoring I can see ‘Health’ is 20 and red, and there is 1 ‘500+ Code’ reporting. Here I would one-click on the ‘500+ Code’ to learn exactly what browser / site is throwing that code, and why. With just another one-click and you will be in full context within Splunk Observability Cloud. That’s right, here you get native results in just 2-clicks and 2 seconds.

^{Figure 1-6. Observability: DevOps SRE Detailed View}

For more information about glass tables, see Overview of the glass table editor in ITSI and a video Getting started with Splunk ITSI Glass Tables.

Service Analyzer, Services, and KPIs in the Content Pack for Splunk Observability Cloud 

The Content Pack for Observability includes 1 Service Analyzer, 25+ Services and 70+ KPI’s which we will highlight each of these here.

Service Analyzer:

The Service Analyzer is the home page for Splunk IT Service Intelligence (ITSI) and serves as your starting point for monitoring your IT operations. The Service Analyzer enables you to see the live health of your IT environment at a glance.

The Service Analyzer provides an overview of ITSI service health scores and KPI search results that are currently trending at the highest severity levels. Use the Service Analyzer to quickly view the status of IT operations and to identify services and KPIs running outside expected norms. Click on any tile in the Service Analyzer to drill down to the deep dives for further analysis and comparison of search results over time.

There are two service analyzer views: the tile view and the tree view. You can drill down to more detailed information from each view to investigate services with poor health scores.

Within this Service Analyzer we are viewing in “Tree View” and can clearly see each of the 3 major Services, and also their ‘status’ with regards to how the underlying KPI’s are reporting

^{Figure 1-7. Observability: Service Analyzer - Tree View}

Services:

A Service is a logical mapping of IT objects that applies to your business goals. The definition of a service is fairly broad. Create business and technical services that model those within your environment. Some services might have dependencies on other services. Services contain KPIs which make it possible to monitor service health, perform root cause analysis, receive alerts and ensure that your IT operations are in compliance with business service-level agreements (SLAs).

Below you will find the ‘Tile View’ of the Observability Cloud Content Pack view in Service Analyzer. As you can see this provides you with insights across all 25+ Services, and to their status for the given time range selected, along with the ability to one-click into more results for any of these Services to see the KPI’s, Entities and more.

^{Figure 1-8.  Observability: Service Analyzer - Tile View}

KPI’s:

A KPI is a recurring saved search that returns the value of an IT performance metric, such as CPU load percentage, memory used percentage, response time, and so on. A KPI is used to monitor the health of a service.

You create a KPI within a specific service. It defines everything needed to generate searches to understand the underlying data, including how to access, aggregate, and qualify with thresholds. You can use the search result values to monitor service health, check the status of IT components, and troubleshoot trends that might indicate an issue with your IT systems.

Within the Content Pack for Splunk Observability Cloud, we include 70+ KPI’s, so you have deep insights across your observability results. As you can see in the below screenshot, access to the results are simply one-click away. You can also quickly see the underlying Entities and how each is reporting.

^{Figure 1-9. Observability: Service Analyzer - KPI’s - Entities}

Entity Types and Vital Metrics in the Content Pack for Splunk Observability Cloud 

Entity Types and Vital Metrics

The Content Pack for Splunk Observability Cloud includes custom entity types. You can use associations to visualize and troubleshoot various entities. For example, this Content Pack ships with an ‘Entity Type’ of “AWS EC2” to import your AWS EC2 services as entities. You can group Entities by Entity Type in the Infrastructure Overview, and enable visualization of key metrics relating to the health of AWS EC2 entities (Services). 

The content pack includes 15 custom Entity Types, one for each of the metrics from the Splunk Infrastructure Monitoring Add-on, one for Splunk APM, and one for each of the metrics from the Splunk Synthetic Monitoring Add-on.

1.  AWS EC2

2.  AWS Lambda

3.  Azure Functions

4.  Azure VM

5.  GCP Cloud Functions

6.  GCP Compute Engine

7.  Kubernetes Pods

8.  OS Hosts

9.  Splunk Infrastructure Monitoring

10.  Splunk APM

11.  Synthetic API

12.  Synthetic Benchmark

13.  Synthetic Content

14.  Synthetic HTTP

15.  Synthetic Real Browser

^{Figure 1-10. Observability: Infrastructure Overview - Entity Types}

Vital Metrics

Within the Content Pack for Splunk Observability Cloud you will receive 34 vital metrics out of the box. These show a critical summary within the Entity Type via a set of Vital Metrics which describe the overall health of entities of that type, including things like: CPU Utilization, Network In, Network Out, Disk Read Ops, Disk Write Ops, and a lot more. You can view these metrics on the Entity Health page and drill down further into individual Exchange entities. 

^{Figure 1-11. Observability: Infrastructure Overview - Vital Metrics}

You can optionally add, modify, or delete the preconfigured entity types. For instructions to create and edit entity types, see Create custom entity types in ITSI. 

Dashboards in Content Pack for Splunk Observability Cloud

A dashboard is used to represent tables or charts which are related to some business meaning. It is done through panels. The panels in a dashboard hold the chart or summarized data in a visually appealing manner. We can add multiple panels, and hence multiple reports and charts to the same dashboard.

Within the Content Pack for Splunk Observability Cloud, we have provided 13 Dashboards to provide easy access to valuable information in a quick and easy manner for you.

1.  SIM - Infrastructure Metrics and Logging

2.  Splunk APM Overview

3.  Splunk Infrastructure Monitoring Command Health Check

4.  Splunk Infrastructure Monitoring Modular Input Health Check

5.  Synthetic API Check Detail

6.  Synthetic Benchmark Check Detail

7.  Synthetic Content Check Detail

8.  Synthetic HTTP Check Detail

9.  Synthetic Monitoring KPI Browser

10.  Synthetic Monitoring KPI Comparison

11.  Synthetic Real Browser Check Detail

12.  Welcome to Splunk Synthetic Monitoring Add-on

13.  SIM Navigation

Below is the Splunk APM Overview dashboard, providing a RED Metrics (Rate, Error, Duration) summary Service as you desire. In this example, we are looking at the “checkout” service.

^{Figure 1-12. Observability: Splunk APM Overview dashboard} 

Next Steps

Details are what I expect, so now you have them….thanks for hanging in there, and I hope you enjoyed all the visuals along the way. If you learn better through watching a video, or simply want more...checkout the most Content Pack for Splunk Observability Cloud Detailed Video. 

Now you know all about the Content Pack for Splunk Observability Cloud, and it is time to install it, and get the value for yourself, now!

For detailed installation steps, see Install and configure the Content Pack for Splunk Observability Cloud.

This blog post was authored by Todd DeCapua, Field Solutions Engineer [IT & Observability] at Splunk with special help from: Tom Martin, Marie Duran, Adam Schalock, Jeremy Hicks, and Joel Schoenberg at Splunk.