Getting Started with the Splunk Distribution of OpenTelemetry Java

Splunk is committed to using inclusive and unbiased language. This blog post might contain terminology that we no longer use. For more information on our updated terminology and our stance on biased language, please visit our blog post. We appreciate your understanding as we work towards making our community more inclusive for everyone.

Splunk Distro for OpenTelemetry is a secure, production-ready, Splunk-supported distribution of the OpenTelemetry project and provides multiple installable packages that automatically instruments your Java application to capture and report distributed traces to Splunk APM (no code changes required!), making it easy to get started with distributed tracing!

The Splunk distribution of OpenTelemetry Java Instrumentation provides a Java Virtual Machine (JVM) agent that automatically instruments your Java application to capture and report distributed traces to Splunk APM.

If you're currently using the SignalFx Java Agent and want to migrate to the Splunk Distribution of OpenTelemetry Java Instrumentation, see Migrate from the SignalFx Java Agent.

This Splunk distribution comes with the following defaults:

• B3 context propagation.
• Jaeger-Thrift exporter configured to send spans to a locally running SignalFx Smart Agent (http://localhost:9080/v1/trace).
• Unlimited default limits for configuration options to support full-fidelity traces.
  
  

🚧  This project is currently in BETA.

Getting Started

To get started, download the JAR for the agent's latest version and add its path to your JVM startup options.

For example, if the runtime parameters were:

$ java -jar target/java-agent-example-1.0-SNAPSHOT-shaded.jar https://google.com

Then the runtime parameters would be updated to:

$ curl -L https://github.com/signalfx/splunk-otel-java/releases/latest/download/splunk-otel-javaagent-all.jar \

    -o splunk-otel-javaagent.jar

$ java -javaagent:./splunk-otel-javaagent.jar -Dotel.resource.attributes=service.name=my-java-app \

    -jar target/java-agent-example-1.0-SNAPSHOT-shaded.jar https://google.com

The -javaagent needs to be run before the -jar file, adding it as a JVM option, not as an application argument. For more information, see the Oracle documentation.

⚠️  Specify the agent as the only JVM agent for your application. Multiple agents may result in unpredictable results, broken instrumentation, and in some cases might crash your application.

To see the Java Agent in action with sample applications, see our examples.

Basic Configuration

The service name resource attribute is the only configuration option that typically needs to be specified. You can set it by adding a service.name attribute as shown in the example above.

A couple of other configuration options that may need to be changed or set are:

• Endpoint if not sending to a locally running Smart Agent with default configuration. See the Jaeger exporter section for more information.
• Environment resource attribute (example: -Dotel.resource.attributes=service.name=my-service,deployment.environment=production) to specify what environment the span originated from.
  
  

Supported Java Versions

The agent works with Java runtimes version 8 and higher and supports all JVM-based languages (for example, Clojure, Groovy, Kotlin, Scala). Supported libraries and versions are listed here.

Advanced Configuration

For the majority of users, the Getting Started section is all you need. The following section contains advanced configuration options. You can find advanced configuration information  here. 

Manually instrument a Java application

Documentation on how to manually instrument a Java application is available here.

To extend the instrumentation with the OpenTelemetry Instrumentation for Java, you have to use a compatible API version. The Splunk distribution of OpenTelemetry Java Instrumentation version 0.9.0 is compatible with the OpenTelemetry Instrumentation for Java version 1.1.0 and API version 1.1.0.

Correlating Traces with Logs

To correlate traces with logs it is possible to add the following metadata from traces to logs:

• Trace: trace_id and span_id
• Resource: service.name and deployment.environment
  
  

Documentation on how to inject trace context into logs is available here.

To log resource context, the Splunk distribution exposes resource attributes as system properties prefixed with otel.resource. which can be used in logger configuration.

Example configuration for log4j pattern:

<PatternLayout>

  <pattern>service: ${sys:otel.resource.service.name}, env: ${sys:otel.resource.environment} %m%n</pattern>

</PatternLayout>

Or logback pattern:

<pattern>service: %property{otel.resource.service.name}, env: %property{otel.resource.environment}: %m

Troubleshooting

To turn on the agent's internal debug logging:

-Dotel.javaagent.debug=true

⚠️  Debug logging is extremely verbose and resource-intensive. Enable debug logging only when needed and disable when done.

And just like that you’re up and running! You can find more information on the docs site.

What’s Next?

OpenTelemetry Javascript v1.0.0 will be GA soon. Check back here or the OpenTelemetry blog for updates. 

ICYMI

• Getting started with OpenTelemetry .NET and OpenTelemetry Java
• Getting Started with OpenTelemetry Python
• Getting Started with the Splunk Distribution of OpenTelemetry Python
  
  

Splunk is a contributor to OpenTelemetry and committed to accelerating the adoption of the project. For more information about Splunk and OpenTelemetry, you can check out our latest OpenTelemetry blog posts.