Executive Lookout: Observing Observability

Splunk Observability is incredibly good at details! Many of us use it as a metaphorical microscope through which we observe our software. But how do you observe the long-term trends and usage of that microscope?

There are numerous organization-level metrics provided in Splunk Observability that can be used to chart organization-level concerns. These can be leveraged in various ways to understand things like uptake, billing and just how much value Observability is providing. In addition, Splunk Observability provides tools for long-term comparisons, which can be leveraged to establish the trending directions of those metrics.

So grab a copy of the free Executive Dashboards available from the Splunk Observability Content Contribution repository on Github and we’ll take a look at how we can get some lowercase observability into our Observability!

High-Level Views: Compare and Contrast

At a high level, directors, CIOs, and other leaders may have some common questions:

• “Who in our organization is using Observability the most?”
• “Is the performance of our software improving compared to last quarter?"
• “Where can we get a quick understanding of what we’re paying for?”
• “Is user experience deteriorating compared to last quarter on specific app versions or browsers?”
• “How can we measure our current usage against our historical usage?”
• “How can we measure uptake of Observability as a tool and solution?”

These sorts of questions can be difficult or have complex answers. But that first step of identifying what you want to know, is often the hardest.

Let's take billing as an example, and work from there. APM and IMM have various billing plans. Each of these plans can be monitored with org-level metrics for things like host count, number of metrics, or traces/data points per minute. 

Having identified the metrics, we can easily chart them in Splunk Observability. But, to unlock high-level understandings, we need to focus on comparing these metrics over the long term. The differences between “What are we doing today?” and “What were we doing last month or last quarter?” provide the guide posts to understanding the success of an organization’s monitoring journey. 

^{Figure 1-1. Executive Dashboard for Billing: Provides 4 week and 12 week comparisons of billable metrics}

For example: Billable metrics important to each of the Splunk Observability plans are shown in the above dashboard as simple tables. Each table compares current usage against 4 and 12 week historicals with change presented as percentages. It may not be imperative to know the exact number of bytes sent 12 weeks ago, but the delta or amount of change can be incredibly important (let’s call it DiffOps for “Buzzword Bingo” purposes!)

What Do These Executive Dashboards Include?

Executive Dashboards are currently available as easily downloadable and usable Terraform files so you can manage them as code (GitOps anyone?) The included dashboards cover a range of Splunk Observability concerns distilled down into 4 dashboards. 

Below are the currently available Executive Dashboards:

1. APM / IMM: Interested in 4 and 12 week trends for your R.E.D./L.E.T.S. Golden Signals? Look no further! Comparisons are provided both organization-wide and broken down by the top and bottom 5 Services.
2. Real User Monitoring (RUM): Want to see long-term trends of what Real Users are doing on various browsers, apps, and app versions? The RUM Executive dashboard provides these breakdowns as 12-week comparisons so you can more easily check up on what your users are experiencing on your web properties and in your apps. 
3. Log Observer: Easily check in on Log Observer (and Log Observer Connect) usage for both normal and Splunk Profiling logs organized by ingest token. The top and bottom 10 tokens are highlighted for convenient comparison. Charts for Events by Severity are also included, along with simple instructions on how to obtain this metric from Log Observer. Curious to know how much of your logging entitlement is being used for INFO or DEBUG logs? Wonder no more!
4. Billing Overview: This view provides trends and comparisons of the “billable metrics” for various Splunk Observability plans. Metrics for all types of plans (Host, TPM, DPM, etc) are represented here with 4- and 12-week comparisons. Useful metrics for Log Observer and RUM usage are also provided.

These views are targeted to give the most complete high-level view of your organization in Splunk Observability. Each dashboard leverages common org-level metrics (with the exception of the single Log Observer Severity metric that can be easily configured) to help you examine everything from User Experience to Billing and beyond!

Check out our Splunk Observability Content GitHub repository and grab these dashboards!

Watching on High Without Having to Try

Because these dashboards are free, publicly available and provided with Terraform configurations, they’re a low-effort easy way to quickly establish trends. A single person can easily deploy the dashboards and manage them as code (think of that whole Monitoring As Code you’ve probably heard so much about).

Want a slightly more detailed breakdown or comparisons for different timespans? Each of the dashboards and charts can be easily modified in Terraform for exactly the view you’d like! Easy configuration, easy results and an eye in the sky to track your Splunk Observability concerns.

Next Steps

Check out the Splunk Observability Community Contribution repository for these free Executive Dashboards along with other innovative contributions from the Splunk community.

If you’re not currently a Splunk Observability customer, but you’re interested in bringing some of the above benefits to your organization’s monitoring journey, sign up to start a free trial of the Splunk Observability Cloud suite of products today!”

This blog post was authored by Jeremy Hicks, Observability Solutions Innovation Engineer at Splunk