Mirrored Dashboards: Efficient Dashboard Management for Enterprise-Scale Monitoring

Imagine a world where every single team and every single service in your organization always has at least a base set of dashboards set up for monitoring, where the charts are standard but the data shown is localized for the specific team or service. Imagine improving such standard dashboards over time by making changes in one place and having them propagate everywhere. Imagine not worrying about changes being made inadvertently, while still enabling teams to customize what they see locally. With the general availability of Mirrored Dashboards in Splunk Infrastructure Monitoring, you can do all of this and more.

Mirrored Dashboards help organizations establish best practices in monitoring by enabling the broad distribution of standard dashboards, keeping them up to date over time, and allowing localized customizations all while keeping proliferation in check. This functionality is the latest in our suite of Enterprise Service Bureau capabilities that enable organizations to efficiently implement monitoring at scale. Mirrored Dashboards fit elegantly with existing Splunk Infrastructure Monitoring functionality around dashboard groups, teams, permissioning, and monitoring as code.

Why Mirror a Dashboard?

Modern enterprises typically need to strike a balance between centrally managed tooling and content, and the ability of individuals and teams to perform self-service tasks and create custom views relevant to them. Doing this well leads an organization to achieve high coverage and high quality of monitoring.

In many large organizations, it is common for a central observability team to create charts and dashboards that are relevant to everyone monitoring a particular type of product or service. Then you have individuals who may want to borrow a dashboard created by a subject matter expert because that dashboard is seen as reflecting best practices. Teams that work on services dependent on one another may want to achieve shared visibility by including each others’ dashboards in their own dashboard groups. Even the same user may desire to persist multiple flavors of the same dashboard – filtered for, say, different combinations of geography and application. Clearly, there are many reasons why users need to see the same dashboard in multiple places, in multiple flavors.

However, as ad-hoc cloning and modification of dashboards becomes common practice, you can end up with a large number of duplicate charts and dashboards that are redundant, outdated, or trivial. These can’t be uniformly updated or easily managed, and create clutter that makes it more difficult to use your monitoring solution.

Mirrored Dashboards help avoid these pitfalls by enabling people to create their own localized views of standard dashboards, and automatically propagating updates made to those dashboards in the future.  

With this new functionality:

1. Mirrors of the same dashboard can be added to multiple dashboard groups (or even multiple times to one dashboard group):
   
   
2. Users can make changes to the dashboard  (e.g. to its charts and layout) anywhere the dashboard appears, as long as the user making changes has write permissions for the dashboard.
   
   
3. Changes made to the dashboard itself propagate to all of its mirrors:
   
   
4. Local customizations (most importantly to filters and variables) can be applied to every mirror, so that the data shown is selected for the specific local context without affecting other mirrors:
   
   
5. Mirror customizations can be edited by anyone with write permissions for the dashboard group in which the mirror appears – write permissions for the dashboard itself are not needed.
    

Walking Through an Example

Consider the following scenarios – they illustrate when it is useful to create mirrors of dashboards:

• Some user in your organization loves one of your dashboards, and wants to view that dashboard without having to go to your personal dashboard group every time. To do so, she adds a mirror of your dashboard to her own user dashboard group.
  
• You are creating standard dashboards for use by teams throughout your organization, and want all teams to see any changes to charts in the dashboard, while also letting their members set dashboard filters and variables to only see data relevant to them.
  
• You regularly look at various Splunk Infrastructure Monitoring built-in dashboards, which are read-only, so you can’t save the dashboards with your choice of filters. You would like to view these dashboards in a single place, with your specified filters, and you would like to automatically get future improvements to the built-in Splunk Infrastructure Monitoring dashboards – so you add mirrors of them to a dashboard group that you use regularly.
   

Adding a Dashboard to Multiple Groups, But with Different Filters Applied

Say you want to add a dashboard representing the health of a Kubernetes cluster to two distinct dashboard groups, Payment and Checkout, but with different variable filters that show Kubernetes metrics for clusters running those specific services. The dashboard in Payment should be filtered for appropriate service (Payment) and namespace (default):

On the other hand, the dashboard in Checkout should be filtered for the Checkout service:

The same dashboard now appears in two different places, with two different values for the Service variable. However, if you were to edit the dashboard from one of these places – say the Payments dashboard group – by replacing one of the charts (Number of Pods per Node) with another (Bytes in per Pod):

The mirror of this dashboard in group Checkout will also show this new chart, but continue to be filtered on different variable values compared to the mirror in group Payment:

The customization options specified when you add a dashboard to a group can subsequently be edited from the Dashboard Info and Dashboard Variables tabs. The Dashboard Info tab also links to all mirrors of the dashboard and lists the groups that they appear in.

As with all things Splunk Infrastructure Monitoring, everything you can do in the UI can also be automated through API. Mirrored Dashboards can and should be used if your team is already defining charts and dashboards with a monitoring-as-code approach.

Enterprise Ready Dashboard Management

With Mirrored Dashboards, enterprise service bureau teams (which may also go by the names observability, platform, or central monitoring teams) can cater to a mix of skills and monitoring needs across teams in their organization. One team may want to create only custom monitoring content starting from scratch, but another may be sufficiently served by adding some customizations on top of a standard dashboard provided by the central team. This balance between flexibility and control is needed in most complex organizations.

Mirrored Dashboards enable a range of powerful use-cases that ensure your users get the right dashboards that they care about with the data that they care about, while also preventing the buildup of redundant and outdated dashboards over time.

Learn more about Splunk Infrastructure Monitoring and download a free 14-day trial. 

Thanks,
Suhail Rashid and Aaron Sun