Full Java Observability in 5 Simple Steps

Java is one of the most popular, flexible and useful programming languages with a very vibrant community to support it. Many of our customers use Java to create amazing applications, it’s an application on a single VM, or based on microservices running on Kubernetes. Naturally, we made it simple to understand the performance of Java-based applications using SignalFx Microservices APM. In this blog post, I will outline the 5 simple steps necessary to enable full observability for a single JVM Java Servlet application written in Java 8. I’ll be using Ubuntu Linux 18.0.4 as an example, but similar steps can be applied to any OS.

Observing Java applications requires monitoring of Java virtual machine metrics via JMX, metrics of the underlying host or system, and traces of the Java application calls correlated in rapid fashion.

Requirements

To try this on your own, you’ll need the following:

• SignalFx μApm and Infrastructure access
• SignalFx API Token
• SignalFx Realm
• Linux Ubuntu or other OS that supports Java 8 with ports 1099, 1999, 8888 available and the ability to:
• Reach SignalFx Agent location
• Clone the following Git Repository
   

Steps to Complete:

1) Install SignalFx Smart Agent

First, we must get our SignalFx API token and Realm from our SignalFx account on www.signalfx.com . Login to your SignalFx account and take note of the realm in the URL once you’re logged in: https://app.us1.signalfx.com/. In this case, our realm is us1.

Next, open the “Person” menu item on the far right and click on Organization Settings -> Access Keys

You can use your default access key for this or create a new one. Expand one of the keys and click on “Show Token” as shown in the green square in the image below:

Now that we have our SignalFx Realm and SignalFx API key, we can install the SignalFx agent. Enter the following commands into your terminal, replacing YOUR_SIGNALFX_REALM and YOUR_SIGNALFX_API_TOKEN with the realm and API token we acquired in step 1 above:

This format indicates Terminal input:

    curl -sSL https://dl.signalfx.com/signalfx-agent.sh > /tmp/signalfx-agent.sh

    sudo sh /tmp/signalfx-agent.sh --realm us1 <YOUR_SignalFX_REALM> <YOUR_SignalFx_TOKEN>

2) Confirm Your SignalFx Smart Agent Installation

In order to confirm our SignalFx agent is correctly installed and sending data to SignalFx we have 2 steps. First you can check the status of the signalfx-agent service as follows:

    sudo signalfx-agent status

You should see output similar to the following:

    splunk@signalfx-java:~$ signalfx-agent status

    SignalFx Agent version:           5.3.3

    Agent uptime:                     45s

    Observers active:                 host

    Active Monitors:                  9

    Configured Monitors:              9

    Discovered Endpoint Count:        6

    Bad Monitor Config:               None

    Global Dimensions:                {host: signalfx-java}

    GlobalSpanTags:                   map[]

    Datapoints sent (last minute):    180

    Datapoints failed (last minute):  0

    Datapoints overwritten (total):   0

    Events Sent (last minute):        16

    Trace Spans Sent (last minute):   0

    Trace Spans overwritten (total):  

Next, let’s look back in our SignalFx account on www.signalfx.com and validate that we are seeing our host and metrics are flowing from our newly installed agent. Once you are in SignalFx click on the “INFRASTRUCTURE” tab at the top and click on Hosts (Smart Agent / collectd ). 

Here we should see our host where we just installed the SignalFx agent. Your infrastructure navigator should look similar to the image below:

3) Configure SignalFx Smart Agent

The SignalFx agent is easily customizable and when you make changes to your configuration the SignalFx agent will automatically restart for you.  All configuration for the SignalFx Smart Agent is done in a configuration file called agent.yaml. In our installation, agent.yaml is located in /etc/signalfx/. 

First, we will acquire examples including an updated agent.yaml (SignalFx Agent configuration) file that monitors JMX, including custom Groovy scripts for calling MBean operations. In this example, we will call an MBean operation that tells us if we have any deadlocked threads in our Java VM.

a. Acquire examples

We will start by Git cloning a sample of the SignalFx Java training module source code:

    sudo git clone https://github.com/shabuhabs/SfxApmTrainingJava.git

Change your directory to the new local repository you just cloned:

    cd SfxApmTrainingJava

b. Java APM and JMX Monitoring Setup

Now, we must make some changes to the default agent.yaml file that the SignalFx Smart Agent installs. In this example, we have a pre-built agent.yaml for you to leverage within the local repository you just cloned.

(NOTE: Assumes port 1999 and port 1099 are available on host and can be connected to from Localhost)

Copy agent.yaml from <local repository>/resources/agent.yaml

    sudo cp ./resources/agent.yaml /etc/signalfx

    sudo cp ./resources/deadlockedThreads.groovy  /etc/signalfx

    sudo service signalfx-agent restart

4) Run Application

We will use Maven to build and deploy our Java application. Our first step is to then install Maven.

    sudo apt-get install maven

To build our application, we simply run a mvn install from the same directory we have been working in (top level of the local repository):

    sudo mvn install

Your results will report BUILD SUCCESS as shown below:

    [INFO] ------------------------------------------------------------------------

    [INFO] BUILD SUCCESS

    [INFO] ------------------------------------------------------------------------

    [INFO] Total time:  01:00 min

    [INFO] Finished at: 2020-07-22T00:15:58Z

    [INFO] ------------------------------------------------------------------------

To run our application, we simply run mvn exec so we create and launch a new Java process:

    sudo mvn exec:exec

5) Validate JMX Metrics and Traces in SignalFx

Generate Traces:
(Assumes port 8888 is available on host)

Open http://<Host where you ran mvn exec:exec>:8888/?amt=100. You should see output similar to the following, depending on your amt=xx requested:

    100 in Great Britain (GBP) is equivalent to CHF118.76 in Switzerland 100 in Great Britain (GBP) is equivalent to EUR110.33 in France 100 in Great Britain (GBP) is equivalent to JPY13,545.76 in Japan 100 in Great Britain (GBP) is equivalent to MYR538.58 in Malaysia 100 in Great Britain (GBP) is equivalent to EUR110.33 in Germany 100 in Singapore (SGD) is equivalent to CHF67.61 in Switzerland 100 in Singapore (SGD) is equivalent to EUR62.81 in France 100 in Singapore (SGD) is equivalent to JPY7,711.34 in Japan 100 in Singapore (SGD) is equivalent to MYR306.61 in Malaysia 100 in Singapore (SGD) is equivalent to EUR62.81 in Germany 100 in India (INR) is equivalent to CHF1.25 in Switzerland 100 in India (INR) is equivalent to EUR1.16 in France 100 in India (INR) is equivalent to JPY142.45 in Japan 100 in India (INR) is equivalent to MYR5.66 in Malaysia 100 in India (INR) is equivalent to EUR1.16 in Germany 100 in United states (USD) is equivalent to CHF94.06 in Switzerland 100 in United states (USD) is equivalent to EUR87.38 in France 100 in United states (USD) is equivalent to JPY10,728.03 in Japan 100 in United states (USD) is equivalent to MYR426.55 in Malaysia 100 in United states (USD) is equivalent to EUR87.38 in Germany 100 in China (CNY) is equivalent to CHF13.45 in Switzerland 100 in China (CNY) is equivalent to EUR12.49 in France 100 in China (CNY) is equivalent to JPY1,533.86 in Japan 100 in China (CNY) is equivalent to MYR60.99 in Malaysia 100 in China (CNY) is equivalent to EUR12.49 in Germany

Log into SignalFx and navigate to the µAPM tab located at the top. This opens the APM monitoring dashboard with all of our services and their respective Requests, Errors, and Duration (RED) metrics and service maps on the right. By understanding the current rate of requests, errors, and durations (aka latency), we know, at a high level, if this service is alive and successfully serving requests in a timely manner. 

Note, all of this is dynamically built seconds after the SignalFx cloud receives your traces. You should see a new service running called “TheNewJavaSerice” as shown in the image below:

That’s it! Our application is now reporting traces to SignalFx. There’s no need for additional instrumentation.

Monitoring the New Service

SignalFx Microservices APM generates an out-of-the-box service dashboard automatically (see below) for you as soon as any service starts reporting traces. Here you can explore more than just the RED metrics (rate, errors, duration) for our Java service. I have access to request rate and latency broken down by endpoint, as well as resource metrics from the underlying host. The idea behind the service dashboard is to: 

• 1) Assess or alert on the health of a service
• 2) Assess or alert on the health of the hardware running the service
  
  

Real World Deadlock Example

Now that our application is instrumented, let’s have some fun! We will intentionally cause a deadlocked thread scenario, which is a commonly monitored MBean Operation via JMX JVM monitoring. Then, we will create a new custom metric and a new chart in SignalFx so we can monitor for deadlocked threads in our VM going forward.

Open http://<Host where you ran mvn exec:exec>:8888/deadlock a few times.

To ensure your threads are deadlocking, simply validate output as shown below in your Terminal window and keep hitting the URL above.

    Alphonse: Gaston has bowed to me!

    Gaston: Alphonse  has bowed back to me!

    Gaston: Alphonse has bowed to me!

    Alphonse: Gaston  has bowed back to me!

    Alphonse: Gaston has bowed to me!

    Gaston: Alphonse  has bowed back to me!

    Gaston: Alphonse has bowed to me!

Validate Deadlock Metrics and Create Charts

Open the Generic Java Stats Dashboard

Navigate to the bottom of the new dashboard you just saved and click on “+ New Chart”.

Type in "jmx.threads", you should see jmx.threads.deadlocked.count auto-complete for you.

Select Single Value Chart

Select “Latest Rollup” under F(x). Note: your number of deadlocked threads may be different than what you see here as it depends on how many times you access the /deadlock endpoint of our Java Application. After validating there are deadlocked threads present, you can “Save and Close” your new chart in the upper right corner.

Name Your New Chart

You should see your new chart now at the top when you click “Dashboards” in the main menu.

Now access http://<Host where you ran mvn exec:exec>:8888/deadlock several more times.

Notice the number of threads has gone up in the top right chart, in this case, from 33 to 41 and the deadlocked thread count went from 4 to 12.

Conclusion

Observing Java applications requires monitoring of Java virtual machine metrics via JMX, metrics of the underlying host or system and traces of the Java application calls correlated in rapid fashion. We were able to do all of the above, and get 1-second to glass monitoring in  5 simple steps. Then, we explored the automatically generated dashboards that SignalFx provided. Finally, we also saw how simple it is to create and visualize a custom metric (Deadlocked thread count), and also create a custom chart for it. Such simplicity and flexibility, combined with our real-time monitoring and full-fidelity tracing enables our customers to focus on their business outcomes instead of struggling to manage legacy APMs!