Enhance Your AIOps Strategy by Utilizing Data Classification

Reduce Noise and Increase Value

Are you looking for a way to increase your AIOps signal to noise ratio and get more value from your data? In this article we will explore how one can utilize OpenTelemetry’s collectors, processors and data models to add or enhance classification attributes. These attributes can help you use your AIOps tools more efficiently and derive more value from your current data.

Splunk AI, Observability, and IT Service Intelligence together can help with data classification and can be a powerful tool for value realization in AIOps. Not all data is of the same value; classification can help enable smarter resource allocation. Classification can be as simple as adding additional attributes which can enable filtering out irrelevant information. AIOps platforms leveraging classification can achieve better data management, improve analysis efficiency and ultimately reduce costs associated with AIOps development and deployment. By combining OpenTelemetry's data collection capabilities and its classification processors one can help optimize what they value in AIOps platforms. 

AIOps platforms, ingest vast amounts of data from a large variety of sources such as:

• Networking Devices
• Business Applications
• Services
• Logs
• A virtually unlimited sources of other data

Early classification of this data can lead to better AIOps outcomes. Classifying data based on observability KPIs such as Severity (critical, major, minor, known bug) and Urgency (immediate action needed, can't wait, etc) will help to allow AI systems such as Splunk AI or Splunk ITSI  to prioritize the most important issues first. The ability to filter on a classifier reduces wasted processing power and human effort spent on less critical events. 

Filter fast by starting at the first point of observability, the OpenTelemetry Collector.  The classification of data can be thought of as a pre-classifier for AIOps algorithms. This means we can structure and organize the data before it is fed into AI algorithms. This structure can include optimization for Observability, Anomaly Detection, Alerting and Event Correlation. Classification helps keep your organization’s priorities in focus through simple things such as classifying data by its severity and urgency.

_{Figure 1-1. This picture displays marine, air and ground transportation. These are examples of logistics related data classification categories.}

Golden Signals or Fool’s Gold?

A grating to screen out large unprocessable rocks before a sluice box can be used to help a miner classify large and small rocks; enabling a more efficient use of resources. Similar to a sluice box an AIOps pipeline works better when you are able to separate signal from noise. The same applies to classifying data. To start this can be based on its Source, Type and the Relevance to a specific task such as security or performance.  An example of a source classification could be network traffic or application logs. The type classification used in Observability might note: informational messages, errors, additional context. These examples help an AIOps engine to filter out what is important to you and your organization. The reduction in noise allows the AIOps engine to focus on the data that truly matters for security, troubleshooting and optimization. 

AIOPs Observability Key OpenTelemetry Features to Leverage:

Standardize Data: OpenTelemetry provides a vendor neutral way of collecting telemetry data from various sources. This standard format makes it easier for classification processors to understand and categorize the data consistently. 

Semantic context: OpenTelemetry allows attaching semantic context in the form of baggage to the data. This context can include things like an application name, service name, and environment. Classification processors can leverage his context to make more informed decisions about data classification.

Classification Processors in the OpenTelemetry Pipeline

Preprocessor Power: OpenTelemetry pipelines support processors that can be inserted before data is sent to the collector classification process. Classification processors such as the attributes processor can be implemented at the service level to analyze and categorize data at an early stage.

Filtering and Routing: Data classification processors can help one filter out irrelevant data or route it to a specific AI art tool depending on its classified importance in purpose. The ability to add filtering and routing reduces the load on downstream systems. These features help to get ingested data to the most appropriate tools for analysis.

Improved Data Quality: Data classification processing can ensure only relevant data reaches AIOps and Observability tools, leading to better quality analysis and more accurate insights.

Faster Anomaly Detection: By filtering out noise, AIOps can focus on anomalies and potential issues more quickly, leading to faster problem resolution.

Enhanced Automation: Data classification can't help automate specific actions based on its data types. For example, critical events could automatically trigger remediation workflows. Splunk SOAR provides Security Orchestration, Automation and Response capabilities to help in automating manual tasks.

Examples of Classification Processor Use-Cases:

• Severity Classification: Assigning severity levels such as critical, major, minor or urgent data, based on predefined rules or machine learning models.
• Component Classification: Categorizing data by its type (hardware, application tier, cloud provider, networking component) for routing to specific Observability and AIOps tools. 
• Anomaly Detection: Identifying unusual data patterns that might indicate potential problems.

AIOps platforms combined with Splunk’s distribution of OpenTelemetry’s data collection and classification processors can help build a more efficient pipeline. Classification is one step in better data management, leading to improved analysis efficiency. Less overhead cost to process data can lead to reduced costs associated with AIOps development, observability and its deployment. 

_{Figure 1-2. This displays an optical breadboard with a DPSS  laser being classified into beams of specific vector angles.}

Data classification can be a powerful tool for reducing costs in AIOps by enabling smarter resource allocation, and filtering out irrelevant information. Confidently being able to reduce the dataset which one is working due to classification is an easy way to improve performance. 

Resource Optimization

Matching Resources to Data Importance: Data classification enables you to allocate different levels of processing power in AI algorithms to different data types for example critical events can be analyzed by more powerful algorithms. This optimizes resource utilization and reduces the overall computational cost of AIOps. Resource matching to allows you to:

• Send less critical events to be handled by simpler algorithms
• Basic Filtering 
• Automate remediation through SOAR playbooks.

Right Sizing Storage Needs: Data classification helps identifying data that can be archived or purged based on its importance and retention requirements. Less critical historical data can be stored in cheaper storage tiers. Critical data requiring real time access can be placed on faster and more expensive storage. This can help reduce the costs for storage which can be associated with AIOps.

Data Classification is an Overall Benefit

Data classification is important when you are implementing AIOps platforms because it can help you become more efficient and cost-effective. Some of the benefits that this translates to include:

• Lowering Data Processing Costs: Classification can lead to less processing power and time wasted on irrelevant data. 
• Reduced Storage Needs: Optimize storage allocation based on the importance of the data. 
• Faster Troubleshooting: Reduce MTTD, MTTA, MTTR, MTTK (mean time to know) through prioritized analysis of critical events. 
• Better utilization of Resources: Matching AI Models, processing power to data and security needs.

Classification is an important step in a successful AIOps journey. OpenTelemetry has some of the key features you would look for in data classification out of the box with classification processors which are beneficial to AIOps. 

Enable Observability, Get a Baseline and then Tune Performance

The early classification of data is an important first step in enabling an effective observability pipeline and in exposing critical signals. Taking a structured approach to categorizing data based on its relevance, potential impact and sensitivity can help an organization prioritize collection of critical metrics. This not only helps with observability and operations, data classification helps to enable better data management, performance, security and anomaly detection. Having well classified data and datasets help to empower organizations and their teams to make better data driven decisions. OpenTelemetry collectors, processors and other tooling can help with not only Observability but also might be able to help clean up your data so you can get insights with ease and spend less time sifting through unimportant data.

Next Steps

Get your applications and services instrumented today with Splunk, Splunk Observability, and OpenTelemetry. Utilize Splunk and OpenTelemetry's technical capabilities to integrate Observability into your AIOps services, continuous delivery, security, and version/process control to promote high levels of software delivery performance. You can learn more by reading An Introductory Use Case Guide: Splunk Artificial Intelligence for Observability. Once you find an interesting use case explore it by signing up to start a free trial of Splunk Observability or the Splunk Platform today!

This blog post was authored by Adam Schalock, Field Solutions Developer at Splunk with special thanks to: Jeremy Hicks and Chad Tripod.