Metrics from Prometheus Exporters Are Now Available with the Splunk Smart Agent

We are excited to announce the expansion of our Smart Agent capabilities with the addition of our direct integration with Prometheus metrics exporters. This extends the range of services our Smart Agent can auto-discover and monitor, and allows customers to monitor Prometheus metrics right alongside data collected from the rest of Splunk's 140+ open-source integrations. Today we’re sharing an overview of the integration, a few technical details, and tips on how you can get started immediately.

About Prometheus

The Prometheus metrics server is a popular open source monitoring system in the cloud-native community, especially among Kubernetes users. It’s simple to deploy and has support for many commonly used applications.

Prometheus takes a pull-based approach to gathering DevOps metrics from hosts and applications. It runs an exporter alongside each service and at regular intervals requests the latest metrics from the service, ingesting them into its timeseries database. Exporters are lightweight HTTP servers that recognize associated services and collect a set of metrics describing the services current state. At times, the service itself exposes an embedded exporter endpoint (such as etcd v3), which eliminates the need for a separate exporter process.

An Open Platform for Prometheus Monitoring

Splunk is committed to providing an open platform for you to easily monitor your systems using a wide range of integrations, without being tied to a proprietary data format. Both our collectd agent and Smart Agent are open source, and we are furthering our commitment to the open source community by joining the Cloud Native Computing Foundation. With our Smart Agent you can now seamlessly monitor your metrics collected through Prometheus alongside all of your other service metrics.

Monitor Prometheus Metrics Using Splunk Infrastructure Monitoring

While Prometheus uses a centralized pull-based model for collecting metrics, the Splunk Smart Agent operates in a push-based model – it emits metrics from services and applications in real-time, directly to Splunk Infrastructure Monitoring. This architecture lets us easily collect metrics from Prometheus exporters running on each host because our agent is pull-based at the host level, and push-based at every level above that.  

By including a new prometheus-exporter monitor, our Smart Agent scrapes metrics from an exporter and sends them to Splunk. Previously, our Metric Proxy could be setup as the gateway to use Splunk as a storage backend for Prometheus. Our new prometheus-exporter monitor cuts out the requirement for going through Prometheus and instead replaces it with our agent.

Configuration

Prometheus exporters lend themselves to our agent’s auto discovery mechanism. Most exporters expose an HTTP port somewhere between 9100 and 9400 and share a common path for getting metrics at /metrics.  As long as there is some other identifying attribute, such as a common pattern for the exporter docker image, or a common name for a port in Docker/Kubernetes, you can create a generic discovery rule such as the following:

monitors:
 - type: prometheus-exporter
   discoveryRule: port >= 9100 && port <= 9500 && container_image =~ "exporter"

Visualizing Prometheus Metrics

Prometheus metrics map easily to our datapoint format, having the same concept of a single metric name, labels (which map 1:1 to our dimensions), and analogous gauges and counters. Prometheus counters correspond to our cumulative counters. Summary and histogram metrics are not currently supported per se in our backend, but it is simple to convert them to individual gauges and counters to send to Splunk. For Prometheus summary metrics, we send each component of the summary (e.g., sample count, sum, quantile values) as a separate gauge with the appropriate metric name and dimension for quantile. Our Prometheus documentation provides detailed information on the conversion process.

Instrumenting Custom Metrics

Splunk Infrastructure Monitoring gives metrics from Prometheus the same capabilities as any other metric in our platform. We allow you to aggregate and correlate your custom metrics without any caveats, monitoring up and down your stack with no limits on your Prometheus metrics or special “custom metric” overage charges. Many other vendors impede developers by charging a premium for custom metrics, throttling their ingest, and limiting how they can be analyzed. Below is an example of using the Prometheus exporter to scrape some kube-state-metrics which you can instantly visualize within your dashboard. It’s just a subset of the metrics you can collect with our Smart Agent. For a full list of the kube-state-metrics visit the GitHub page.

Start Monitoring Prometheus Metrics in Splunk

Prometheus has a long list of exporters that expands the scope of services that our Smart Agent can monitor out-of-the-box. It is especially useful in cases where applications expose Prometheus metrics directly, but is also useful in environments where Prometheus is already in use and the metrics exposed by the exporters have proven sufficient for monitoring a particular service.

The enhanced Prometheus integration furthers our efforts to improve and expand our 140+ open source integrations and built in dashboards for critical open source technologies in the cloud-native software stack. Be on the lookout for more to come!

Get visibility into your entire stack today with a free 14-day trial of Splunk Infrastructure Monitoring.

Thanks,
Ryan Powers