Harnessing the Power of Splunk APM Business Workflows in IT Service Intelligence

As an Observability Strategist at Splunk, I have the unique privilege of partnering with a diverse range of Splunk customers across various industries. This partnership offers me a deeper insight into their essential Observability use cases, how they are utilizing Splunk’s Observability solutions, and their specific needs to maximize both value and efficiency in their Observability practices. In these collaborations, a common theme has emerged: As businesses increasingly rely on complex hybrid and cloud environments to deliver prized customer experiences, the need for a comprehensive, integrated monitoring solution has never been more critical. Navigating these complex and often ephemeral environments presents a significant challenge in understanding application performance and its impact on business goals. This is where Splunk Observability steps in with Splunk APM and Splunk IT Service Intelligence (ITSI). 

One of my favorite features in Splunk APM are the dynamically generated service maps. These service maps offer DevOps teams immediate insights into service interactions, dependencies, and performance – a real game-changer for troubleshooting. Moreover, the metrics used in Splunk APM for troubleshooting provide crucial context for business executives and service owners about how vital business processes are performing. With Splunk APM Business Workflows, you get even more precise visibility into specific business transactions or processes. This not only streamlines troubleshooting for engineers but also offers business-relevant insights.

One of the most difficult challenges service owners face is understanding how application health and performance impact the business processes that they are designed to deliver. Splunk IT Service Intelligence (ITSI) provides cross-silo visibility into the health and key performance indicators of IT services. By integrating data across the enterprise, Splunk ITSI enables you to visually map services and key performance indicators (KPIs) to discover new insights, baseline normal operations using machine learning and translate operational data into business impact. The key word here being business impact. 

Many customers already find tremendous value in using Splunk APM data in ITSI through the ITSI Content Pack for Splunk Observability Cloud, available in the Splunk App for Content Packs, which has enhanced how users visualize and manage the health of applications within ITSI. With the integration of key components such as Splunk Infrastructure Monitoring, Splunk APM, Splunk Synthetic Monitoring, and Splunk Real User Monitoring (RUM), the Content Pack for Splunk Observability Cloud offers a holistic view of enterprise health. This blend of pre-built ITSI content ensures that both web-based and mobile applications, along with cloud infrastructure, can effectively be monitored alongside and correlated with existing ITSI services, business focused KPIs, and be used to enrich events analyzed during automated event aggregation.

Splunk APM and ITSI both offer service dependency mapping, each with a unique focus. Splunk APM's service maps are tailored for problem isolation, troubleshooting, and debugging, helping teams quickly address issues. On the other hand, ITSI's service maps provide a broader perspective, offering insights into business process dependencies across diverse data sources and translating operational data into tangible business impact. A popular request from customers utilizing both ITSI and Splunk Observability is the ability to generate ITSI services directly from Splunk APM's service topologies. This integration deepens the understanding of service interdependencies and clarifies the influence of application performance on business results.

I'm thrilled to share that the latest release of the ITSI Content Pack for Splunk Observability Cloud now enables the automatic creation of ITSI services using service topologies from Splunk APM Business Workflows.

This integration lets users select Business Workflows configured in Splunk APM and automatically create ITSI Services with corresponding KPIs. This significantly reduces the time needed to leverage Business Workflow metrics in ITSI. Plus, this new feature supports ITSI Service Sandboxes, introduced at .conf23, which lets users map services directly in the ITSI UI. This feature is fantastic in a pre-production setting, where you can add, manage, and tweak services, link dependencies, and collaborate with teams before rolling out to production.

Business Workflows are perfect for ITSI because the metrics they offer provide a window into application performance while focusing specifically on mission-critical business transactions or processes. Let's look at some examples of Business Workflows, why they're essential, and who benefits from them.

Understanding Business Workflows

Confidently delivering services in the complex landscape of hybrid and cloud computing requires a clear understanding of your key business transactions from start to finish. This is where Business Workflows come into play. A Business Workflow represents the complete path taken by a specific activity or transaction. Think of it as a detailed map of a digital journey, honing in on the elements that are most critical to your business's success. This focused approach is instrumental in cutting through the clutter and zeroing in on what truly matters for your operational efficiency.

Examples of Business Workflows

Business and engineering executives might use Business Workflows in Splunk APM to:

• Monitor end-to-end rate, error, and duration (RED) metrics for most critical transactions
• Ensure the organization is meeting associated KPIs, such as
  • What is the 90th percentile duration for users logging in?
  • What percentage of checkout requests are failing?

Site reliability engineers and service owners might use Business Workflows in Splunk APM to:

• Better prioritize, understand blast radius, and troubleshoot when issues occur
• Narrow down context to only services involved in the transaction of interest 
  • Which downstream service is responsible for the increased response time of the API class?
  • Are the errors in the Login service only happening for API calls from the homepage?

Why Embrace Business Workflows?

1. Holistic Monitoring of Business Transactions: Business Workflows improve the way business transactions are managed. By automatically grouping various microservices that constitute a business transaction, these workflows enable comprehensive monitoring of KPIs for these transactions. This comprehensive view is crucial for pinpointing root causes and resolving bottlenecks in your services efficiently.
2. Achieving SLAs and SLOs with Precision: For business and engineering leaders, Business Workflows are more than just tools – they are strategic assets. They aid in tracking vital metrics such as rate, error, and duration (RED) for business transactions. This detailed monitoring ensures that Service Level Agreements (SLAs) and Service Level Objectives (SLOs) are not just met, but consistently exceeded, aligning operational performance with business goals.
3. Streamlined Troubleshooting: When it comes to troubleshooting, site reliability engineers and service owners find Business Workflows indispensable. These workflows allow for a focused approach to problem-solving, enabling engineers to concentrate solely on the services directly involved in a specific transaction. This targeted method leads to prioritized and faster resolutions, ensuring that service reliability is maintained at the highest level.

Each Business Workflow generates Monitoring and Troubleshooting MetricSets for traces that fit Business Workflow rules. This means you are able to track request rate, error rate, and duration (latency) for the entire Business Workflow, not just at the individual service level. Which makes these metrics perfect candidates for ITSI Service KPIs.

In this example, a Business Workflow has been configured to monitor the checkout process for an online retail store, which is a critical business transaction. Using the Business Workflow, only the services that are involved in the checkout process are focused, eliminating noise and minimizing complexity. This allows SREs and service owners to not only understand that the payment service has a high error rate (as indicated by the giant red circle in the service map), but also these errors are directly affecting the checkout process.

Additionally, the Monitoring MetricSets provided by the Business Workflow allows business and engineering teams to answer questions such as:

• How many checkouts were executed in the last 24 hours?
• What percentage of checkouts are failing?
• How long does the checkout process take?

Hopefully, by now the power of Business Workflows and why the Monitoring MetricSets they generate provide value to business and engineering teams is clear. For this reason, the latest version of the ITSI Content Pack for Splunk Observability Cloud allows you to import entire Business Workflows topologies with just a few clicks. 

By doing so, the service dependencies of a Business Workflow configured in Splunk APM are used to automatically create ITSI services. Using Service Templates, each service created in ITSI has pre-defined KPIs which are automatically assigned and focus on RED metrics at the Business Workflow level as well as for each individual service. This eliminates all of the legwork needed to untap the value of Splunk APM Business Workflow metrics in ITSI. Furthermore, using Service Sandboxes in ITSI allows you to modify the Business Workflow services before publishing them so they are tailored to provide the most relevant and actionable insights into business processes.

As we've explored, the power of Business Workflows in Splunk APM, coupled with the insights of IT Service Intelligence, creates a formidable toolset for understanding and optimizing your IT operations. Whether you're an engineer troubleshooting complex issues or a business leader seeking clarity on how IT impacts your critical business processes, this integration offers invaluable insights.

The latest release of the ITSI Content Pack for Splunk Observability Cloud marks a significant milestone. It brings the much-anticipated capability of automatically creating ITSI services using the service topologies from Splunk APM Business Workflows. This feature not only saves time but also brings a new level of precision and depth to your service monitoring and troubleshooting efforts.

The integration of these powerful tools provides a comprehensive view of your IT landscape, translating complex data into actionable insights. This clarity is essential in today's fast-paced, digital-first business environment. By leveraging the combined strengths of Splunk APM and ITSI, you can ensure your IT operations are aligned with your business objectives, delivering exceptional experiences to your customers.

I understand that seeing is believing. Therefore, I invite you to experience the full capabilities of this integration firsthand.

• For Existing Users: Upgrade to the latest version of the ITSI Content Pack for Splunk Observability Cloud, available in the Splunk App for Content Packs. Discover how this enhanced integration can transform your IT operations and business decision-making.
• For New Users: If you're not yet using Splunk Observability Cloud, now is the perfect time to start. Sign up for a free trial and begin your journey to more effective, insight-driven IT service management.