How to Instrument a Java App Running in Amazon EKS

As we start to see big moves from monolith deployments to microservices, the adoption of Kubernetes has become top of mind for many SREs. Organizations can leverage the open-source system to automate deployments, scale, and manage containers, making Kubernetes one of the primary solutions for delivering workloads. However, maintaining the system can be difficult and, in some cases, overwhelming. For that reason, it is easy to find cloud platforms offering Kubernetes as a managed service like Amazon EKS. Amazon EKS helps deliver highly available and secure Kubernetes clusters in a way that is both fast and simple. 

Does Amazon EKS Change How I Instrument Applications? 

The short answer is no. Splunk APM makes it easy to leverage the OpenTelemetry collector to collect application metrics with no wasted efforts with future-proof, vendor agnostic instrumentation designed to scale. 

Instrumenting applications running in EKS is very similar to instrumenting applications that aren’t running in EKS, so we’ll demonstrate how to instrument a non-EKS app first, then explain the changes needed if your app runs in EKS. 

Deployment of a Basic Java Application Without Kubernetes

In this example, we will use a basic Java application that performs HTTP GET requests to a website of your choice. To begin, navigate to the Data Setup wizard from the hamburger menu, from the APM Instrumentation menu select Java, and click Add Connection. 

Provide the following details about the environment where the application is deployed in the Configure Integration section.

• Service - The name of the service you are instrumenting
• Endpoint - This setting reflects the endpoint with the Splunk OpenTelemetry Connector installed. By default, the setting is set to http://localhost:4317. This can easily be changed to reflect a remote machine hosting the Splunk OpenTelemetry Connector.
• Environment - The name for the environment that you are instrumenting. 
• Collect Metrics - (Experimental) If enabled, gathers additional JVM-specific metrics.
• Kubernetes - If the application is running within Kubernetes.Legacy 
• Smart Agent - Whether or not you would like to use the legacy SignalFx agent. 
  
  

After the configuration steps are complete, the next step is to install the Splunk OpenTelemetry Connector if you have not already done so. In this example, we have installed the collector on an Ubuntu server running the Java application. To verify the Splunk OpenTelemtery Connector was installed correctly, type the following to ensure the service is running:

service splunk-otel-collector status

Note the splunk-otel-collector is active (running).

The last step is to install the integration by downloading the latest version of the Splunk Distribution of OpenTelemetry for Java with the provided curl command to add to your runtime command. Set the environment variables that provide details about your deployment, automatically created based on the information provided in the previous step. 

Once set, you can then update your runtime command to include the Splunk Distribution of OpenTelemetry Java. Here is my example below of how I updated my runtime command in a non-EKS (Kubernetes) deployment.

java -javaagent:./splunk-otel-javaagent.jar -jar curlapp-1.1-SNAPSHOT.jar

You can also add the environment variables to your runtime command manually. Here is an example. 

java -Dexec.executable="java" -Dotel.resource.attributes=service.name=httpGoogleChecker-bm,deployment.environment=CheckingGoogleBM -Dsplunk.metrics.enabled=false -javaagent:splunk-otel-javaagent.jar -jar curlapp-1.1-SNAPSHOT.jar

Now that you have started your Java application, the data setup wizard will direct you to interact with your data by clicking Explore Service Spans. 

Note the environment name (CheckingGoogleBM) highlighted in the screenshot above. 

How Does the Process Change with Amazon EKS? 

When instrumenting your Java application with Amazon EKS, the process is similar to the one previously described and only changes slightly due to how applications are deployed with Kubernetes. To add, this process will also work the same when deploying your application to a Kubernetes cluster outside of Amazon EKS. To begin instrumenting your containerized Java applications with Amazon EKS, follow the same steps to navigate to the data setup wizard. On the Configure Integration section, select Yes for Kubernetes. 

Click next to proceed to the Install Integration section. The first step is installing the Splunk OpenTelemetry Connector on your Kubernetes cluster if not already done so. To verify if the Splunk OpenTelemetry Connector is installed on your Amazon EKS cluster, update the kubeconfig file for your Amazon EKS cluster and run the following command.

kubectl get pods

The output should show both the agent and cluster receiver pods.

Once installed, use the curl command to download the splunk-otel-javaagent.jar to include with your runtime command within your containerized application. 

Here is an example of my Dockerfile, which copies the splunk-otel-javaagent.jar of the Java application container and executes the application within the run.sh bash script. 

FROM openjdk:8-jre-alpine3.9

COPY target/curlapp-1.1-SNAPSHOT.jar /curlapp-1.1-SNAPSHOT.jar

COPY splunk-otel-javaagent.jar /splunk-otel-javaagent.jar

COPY run.sh /run.sh

ENTRYPOINT ["sh", "/run.sh"]

Contents of run.sh:

java -javaagent:/splunk-otel-javaagent.jar -jar /curlapp-1.1-SNAPSHOT.jar

Next, use the provided example to update your deployment file to inject the environment variables necessary to auto instrument your application for your Java application.

Here is an example of my deployment file updated with the required environment variables in italics. 

apiVersion: v1

kind: Pod

metadata:

  name: httpgooglechecker

spec:

  containers:

  - name: httpgooglechecker

    image: docker.io/astro7982/curlappstatic:lastest

    env:

    - name: SPLUNK_OTEL_AGENT

      valueFrom:

        fieldRef:

          fieldPath: status.hostIP

    - name: OTEL_RESOURCE_ATTRIBUTES

      value: service.name=httpGoogleChecker-k8s,deployment.environment=CheckingGoogleK8S

    - name: SPLUNK_METRICS_ENDPOINT

      value: http://$(SPLUNK_OTEL_AGENT):9943

    - name: OTEL_EXPORTER_OTLP_ENDPOINT

      value: http://$(SPLUNK_OTEL_AGENT):4317

    command: ["/bin/sh"]

    args: ["run.sh"]

Now that all installation requirements are completed create your Java application pod using kubectl create, and within 90 seconds, Splunk APM will begin capturing all traces from your application. 

Note both the service name and environment are reflected in the ones entered in your Kubernetes configuration file.

Now that the service is instrumented, you can see RED metrics created automatically, and you can also view every single transaction’s full trace through your infrastructure. Check out our demo video to see more of what can be done once your application is instrumented. 

Want to try this yourself? Feel free to – you can sign up to start a free trial of the suite of products – from Infrastructure Monitoring and APM to Real User Monitoring and Log Observer. Get a real-time view of your infrastructure and start solving problems with your microservices faster today.

----------------------------------------------------
Thanks!
Johnathan Campos