Introducing Splunk OpenTelemetry Java Lambda Wrapper

AWS Lambda has become a core technology in the shift to cloud-native application development, eliminating infrastructure management and fixed costs. But there are trade-offs with serverless environments. Not having access to the production infrastructure can make debugging difficult and there are a lot of moving parts, adding distributed complexity. Monitoring serverless functions in production requires observability beyond CloudWatch logs and metrics. End-to-end, low-maintenance distributed tracing is essential to understand performance issues and troubleshoot cloud-native application development. This is why we’re excited to announce Splunk OpenTelemetry Java Lambda Wrapper is available in beta! 

OpenTelemetry Java Lambda Wrappers enable frictionless, automatic instrumentation, and support incoming and outgoing context propagation using open standards such as B3. Splunk OpenTelemetry Java Wrappers are distributed as AWS Lambda layers which are published and hosted by Splunk in all AWS regions — making integration fast and easy.

Splunk OpenTelemetry Java Lambda Wrappers add value compared to general OTEL wrappers by providing:

• Zero-code integration (by env variables / configuration only)
• Splunk-centric default configuration values
• Easy ingestion into SplunkAPM
• Hosting as layers
   

Using OpenTelemetry Java Lambda Wrapper to Instrument Your AWS Lambdas

The Splunk OpenTelemetry Java Lambda Wrapper is a distribution of the wrappers in the OpenTelemetry AWS Lambda Instrumentation that enables you to export spans from an AWS Lambda function with Java to Splunk APM without any code changes to your Lambda functions. You can choose between two ways to send your data to us:

1.  Directly from the Lambda wrapper to the Splunk Observability Suite
2. To a Splunk OpenTelemetry Collector that's running within your environment
    

There are two options to use the Splunk Lambda wrapper:

1. Use a Lambda function wrapper directly
2. Use a Lambda layer that Splunk hosts
    

Splunk also provides a Serverless Application Model (SAM) template for deploying the Lambda wrapper with a Lambda handler or a Lambda layer. You can find more information and examples on the GitHub page for Splunk Open Telemetry Java Lambda Wrapper. In this blog, we’ll show you how to deploy the wrapper with a Lambda layer for frictionless instrumentation.

Deploy the Wrapper with a Lambda Layer

Add a layer that includes the Splunk Lambda wrapper to your Lambda function. A layer is code and other content that you can run without including it in your deployment package. Splunk provides layers in all supported regions you can freely use.

You can also deploy the layer with a SAM template. 

To reduce the size of the deployment package, make sure that your Lambda artifact doesn't contain the wrapper.

Follow these steps to configure a Splunk Lambda wrapper to export spans to Splunk APM with a layer that Splunk provides.

1. From the AWS console, add a layer to your Lambda function code.
2. To add a layer that Splunk provides, specify an available ARN, depending on your region. For an available ARN, see the latest available versions of SignalFx Lambda wrapper layers.
3. Verify that dependencies in the layer aren't also in the Lambda function .jar file.
4. Deploy your Lambda function code.
5. Set a wrapper class as the handler for your Lambda function. These wrappers are available:
    

Set the OTEL_INSTRUMENTATION_AWS_LAMBDA_HANDLER environment variable in your Lambda function code:

OTEL_INSTRUMENTATION_AWS_LAMBDA_HANDLER="package.ClassName::methodName"

By default, the Splunk Lambda wrapper uses B3 context propagation. If you want to change this, set the OTEL_PROPAGATORS environment variable in your Lambda function code. For more information about available context propagators, see the Propagator settings for the OpenTelemetry Java Instrumentation.

By default, the Splunk Lambda wrapper uses a jaeger-thrift exporter to send traces to Splunk APM. If you want to use this exporter, set these environment variables in your Lambda function code:

OTEL_EXPORTER_JAEGER_ENDPOINT="http://yourEndpoint:9080/v1/trace"

OTEL_EXPORTER_JAEGER_SERVICE_NAME="serviceName"

SIGNALFX_AUTH_TOKEN="orgAccessToken"

Also, you can set span flush wait timeout, that is max time the function will wait for the spans to be ingested by the Splunk APM. Default is 1 second. Timeout is controlled with a following property (value in milliseconds):

OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_TIMEOUT: 30000

If you want to use a different exporter, set the OTEL_EXPORTERS environment variable. Other exporters have their own configuration settings. For more information, see the OpenTelemetry Instrumentation for Java on GitHub.

Set the environment in Splunk APM for the service with the OTEL_RESOURCE_ATTRIBUTES environment variable:

OTEL_RESOURCE_ATTRIBUTES="environment=yourEnvironment"

Save your settings and call the Lambda function.

AWS Span Tags the Wrapper Adds to Trace Data

The Splunk Lambda wrapper automatically adds span tags to trace data it exports. These are the available span tags for AWS metadata. For more information, see the OpenTelemetry Specification.

Logging

You’ll need to set environment variables for the Lambda function. These environment variables control logging:

Use a Lambda Function Wrapper Directly

You also have the option to add the wrapper as a dependency to the Lambda function. A Splunk Lambda wrapper wraps around an existing AWS Lambda Java function handler. This approach doesn't require any code changes to your Lambda function. When you deploy the Lambda wrapper with a Lambda handler, you add it as a dependency to your Lambda function. Whenever the Lambda function is invoked, it runs the Lambda wrapper which in turn calls your code. You can find more information about how to deploy via this method in Github.

Conclusion

After successfully completing the setup each invocation of your Lambda should create a single span with all appropriate tags, delivered directly to your APM dashboard, tied with other spans in your trace. You can find more information and examples in our docs on Github. Get started today!

Additional Resources

• OpenTelemetry Instrumentation for Java
• OpenTelemetry Java SDK and extensions