What’s New in OpenTelemetry: Community, Distributions, and Roadmap

I am honored to be able to talk about Splunk’s investment and commitment to the OpenTelemetry project. I would like to take this opportunity to talk about the latest in the OpenTelemetry community, as well as the instrumentation and data collection distributions available from Splunk. Be sure to read through the whole post, as you will find some roadmap information too!

OpenTelemetry

If you missed the news, OpenTelemetry — the second most active project in CNCF — has achieved incubation status! This means it is more than ready for you to adopt its “batteries included” implementations. Over the last year, OpenTelemetry has released a stable trace specification as well as stable instrumentation libraries (API + SDK) for the tracing signal in multiple languages, including:

• .NET
• C++
• Java (automatic instrumentation too!)
• JavaScript (automatic instrumentation too!)
• Go
• Python (automatic instrumentation too!l)
  
  

Note: Both Erlang and Ruby should be announcing 1.0 releases soon.

Beyond the tracing signal, the metric data model and API are stable. This paves the way for instrumentation libraries to add native OpenTelemetry metric support in the coming months which includes OpenMetrics compatibility. In addition, the collector released a stable core version for the tracing signal centered around configuration and an OTLP receiver/exporter. Finally, several donations to the project are helping accelerate the roadmap and increase the project’s ability to collect observability data:

• Splunk’s donation of the Flowmill (eBPF) agent
• ObservIQ’s donation of the Stanza (logging) agent
• Google’s donation of the Sqlcommenter (database visibility per language)
  
  

Across the OpenTelemetry repositories, Splunk has been actively helping accelerate the maturity and adoption of the project. Splunk has multiple contributors, approvers, and maintainers across many of the OpenTelemetry repositories. Splunk also offers distributions of OpenTelemetry making it easy to get started with Splunk products, including Splunk Observability Cloud. Distributions are 100% OpenTelemetry compatible, offering a pre-packaged solution that includes the components necessary to work  with Splunk products and enterprise support from Splunk. Let’s drill into Splunk’s distributions and enhancements announced over the last year.

Instrumentation Tracing Updates

Over the last year, Splunk has been helping OpenTelemetry instrumentation libraries hit their 1.0 milestones. In addition, Splunk has been upstreaming capabilities found in its SignalFx instrumentation. Splunk now offers GA versions of the following instrumentation libraries:

• Splunk Distribution of OpenTelemetry Java
• Splunk Distribution of OpenTelemetry Python
  
  

With the GA release of the above libraries, we are also announcing the end of support date for the SignalFx Java and Python tracing (not metrics) libraries which were already deprecated. The end of support date is December 17th, 2022 as stated in Splunk’s documentation. Last year, we announced going all-in in OpenTelemetry and this milestone reinforces our commitment. Of course, we are not done. Over the next several months, we plan to announce GA distributions of C++, Go, JavaScript and Ruby.

Then there is serverless. OpenTelemetry offers AWS Lambda wrapper support for .NET, Java, JavaScipt, and Python today. Splunk provides a distribution for Java (with SignalFx lambda support for traces and metrics in JavaScript, Python and Ruby) and recently released a cross-language wrapper project to improve usability even further. Longer term, we are planning to provide an extension to deploy the OpenTelemetry Collector.

Finally, Splunk just announced profiling support. The Splunk Distribution of OpenTelemetry Java makes it easy to continuously profile your Java applications with minimal overhead. This capability will be extended to more languages in the future.

Instrumentation Metrics Updates

You may recall that one of the benefits of OpenTelemetry instrumentation libraries is that a single implementation per language can offer both tracing and metrics capability. While the community awaits native OpenTelemetry metric support, Splunk is excited to announce automatic runtime metric instrumentation in the Splunk Distribution of OpenTelemetry Java, compliments of Micrometer. Users now have the choice of automatic runtime metric instrumentation or JMX metric gathering from the Splunk OpenTelemetry Connector, making it easier than ever to collect runtime metrics. Expect to see similar automatic runtime metric support for JavaScript and .NETcoming next. Of course, we are 100% committed to offering native OpenTelemetry metric support once it is available.

Instrumentation RUM Updates

Last year, we announced Splunk RUM — the industry’s first full-fidelity, real-user monitoring solution — which focused on browser observability. This year we are excited to announce mobile RUM observability. To support this, we offer the following OpenTelemetry distributions now in beta:

• Splunk Distribution of OpenTelemetry Android
• Splunk Distribution of OpenTelemetry for Swift (iOS)
  
  

Like all OpenTelemetry distributions Splunk offers, these distributions are 100% OpenTelemetry compatible. We are also actively working with the community to help define RUM semantic conventions in the OpenTelemetry specification ensuring data portability for end-users. Be sure to take a look at the RUM OTEP and join the SIG meeting.

Data Collection Updates

On the data collection side, we have been very busy helping with the initial stable release of the OpenTelemetry Collector for the tracing signal. This is a critical milestone for the OpenTelemetry project as it lays the foundation of stable configuration and the initial core components, including the OTLP receiver and exporter. As we announced last year, Splunk also offers a distribution of the OpenTelemetry Collector. Over the last year, we have expanded the packaging options for the Splunk OpenTelemetry Connector offering support for a variety of environments, including:

• Linux (Debian, Red Hat, SuSE on AMD and ARM systems)
• Windows (10 Home/Pro, Server 2012+)
• Mac OS (Intel, M1)
• Docker
• Kubernetes and OpenShift (Operator, Helm, YAML for Linux and Windows)
• Heroku (Buildpack)
• Configuration Management (Ansible, Puppet)
• Amazon (ECS/EC2, Fargate)
  
  

Beyond packaging, we continue to upstream the SignalFx Smart Agent monitors into OpenTelemetry. The Splunk distribution leverages the hostmetrics, k8s cluster and kubletstats receivers in OpenTelemetry. In addition, the Docker, ECS, Host and K8s observer extensions are also leveraged — all of these concepts originated from the Smart Agent and have since been ported to OpenTelemetry (with contributions from Google, Amazon, and others too!). Over the next several months, we plan to migrate even more monitors to OpenTelemetry with Redis on the short-term list.

The recent upstream of the Docker observer marks one of the last major parity gaps between the SignalFx Smart Agent and the OpenTelemetry Collector. To ease the transition, we introduced a migration utility that will assist in converting a Smart Agent configuration into an OpenTelemetry Collector configuration. As a result of this milestone, we are announcing the end of support date for the already deprecated SignalFx Smart Agent. The end of support date is December 17th, 2022 as stated in Splunk’s documentation.

Data Collection Enhancements

In terms of new functionality, earlier this year, we introduced dynamic remote configuration for the Splunk OpenTelemetry Connector, including the ability to provide configuration via environment variables, files, Hashicorp Vault, and Apache Zookeeper with live configuration reload. Now that the initial stable release of the Collector is available, we will continue to upstream this capability so that everyone can benefit from it.

The Splunk distribution offers support for traces, metrics, and logs. Last year we announced log support via Fluentd. Earlier this year we were excited by the donation announcement of ObserveIQ’s Stanza log agent to OpenTelemetry. The integration has been going great, and we are excited to announce that the Splunk OpenTelemetry Connector for Kubernetes now offers native log support without Fluentd. In addition, we have enhanced the chart to fully support both Splunk Cloud and Splunk Observability Cloud — the next major release will mark the replacement of Splunk Connect for Kubernetes, which was based on Fluentd.

We are 100% committed to OpenTelemetry and working to consolidate on the OpenTelemetry Collector as our primary agent. To that end, we are also excited to announce the donation of the Flowmill eBPF agent to the OpenTelemetry project making it easier than ever to get observability data. The eBPF data can currently be sent via the OpenTelemetry log signal. We look forward to sharing more information about Network Performance Monitoring soon!

Looking Ahead

As you can see, Splunk is all-in on OpenTelemetry. Not only are we one of the top contributors to the project, but we are also actively adopting and consolidating on the project. As we look to what’s next, it’s making it even easier to deploy instrumentation as well as providing agent management capabilities. In fact, we are excited to share a public roadmap of capabilities we are planning to invest in located here. We would love your feedback on what you would like to see, so please feel free to file a GitHub issue in any Splunk OpenTelemetry distribution repository.

The future is bright for OpenTelemetry, and we are glad to be part of the journey and incredible community. Please join the conversation on the CNCF Slack, join a SIG, or pick up a GitHub issue with either a help-wanted or good-first-issue label. And of course, be sure to try a free demo of Splunk Observability Cloud — we know you are going to love it!

More Ways to Learn About OpenTelemtry

Want to learn more about how you can start using OpenTelemetry? To learn more about how to capture, generate and export telemetry data into observability tools for comprehensive software performance and behavior analysis, check out the talks below from .conf21 here.

• DVO1500C: Getting Started With the OpenTelemetry Collector  
• DVO1537B: OpenTelemetry at Splunk and Google: How we use it, how our customers use it, and how you can use it. 
• DVO1377B: Observability From The Ground Up: Using OpenTelemetry and Splunk APM at Care.com
• DVO1498C: OTel Me Why
• DVO1179B: Progressive's Journey to Observability; From Hand-Rolled To Shrink-Wrapped