The Extensibility of OpenMessaging Benchmarks

Let's be honest – performance benchmarks in the software industry typically range somewhere between unhelpful and downright misleading. Far too often, we've seen companies rushing to sell tools and platforms, including pub-sub messaging systems, by cherry-picking workloads to showcase a specific system's strengths or focusing on weird edge cases rather than realistic scenarios. We've seen this happen in benchmarks in a variety of domains, from databases to cloud services to hardware. We think that the industry deserves better, especially in the increasingly crucial messaging space. And so we're excited that the OpenMessaging Project, of which Streamlio is a founding member, is stepping up to forge a bold new path that has the potential to change how we approach benchmarks.

Enter OpenMessaging Benchmarks

The OpenMessaging Project is an effort to develop an open standard for messaging systems. Streamlio is a founding member of the project, alongside Yahoo!, Alibaba, DiDi, and others. The OpenMessaging benchmarks project was created as a sub-project under the auspices of the broader OpenMessaging Project. The benchmarks project operates in the same spirit as the umbrella project, and we've been active from the very beginning.

All the code for the OpenMessaging benchmarks project is available on GitHub, in the openmessaging/openmessaging-benchmark repo.

We've been active in the creation of the OpenMessaging benchmarks project. We talked a bit about that in a previous post.

We're extremely excited about the OpenMessaging benchmarks project because it's simply a better approach to benchmarks in just about every way we can think of.

Core Goals

The OpenMessaging benchmarks project is rooted in a small set of goals, all of which seek to:

In this post, we'll focus mostly on realism and extensibility and show you how to create your own benchmarking workloads.

The Importance of Realistic Workloads

The current OpenMessaging benchmark workloads were designed with realism in mind. Workload configurations can vary along a number of crucial axes:

The goal of realism goes hand in hand with the goal of extensibility. In order to run workloads that are realistic for you and your use cases you need to be able to quickly create new ones and even, potentially, contribute them to the project. We'll show you how to do precisely that in the next section.

Creating Your Own Workloads

Workloads are really the heart of all benchmarks, regardless of which type of system you're dealing with. In the OpenMessaging benchmarks project, creating new workloads actually doesn't require you to write code at all. Instead, all you need to do is create a configuration file using YAML and specify some values and you have a new workload. The current workload configurations are here.

When you create a new workload, you can run it on your own or, even better, submit a pull request to the repo and have it added permanently.

First, let's have a look at an existing workload:

name: 1 topic / 1 partition / 1Kb
topics: 1
partitionsPerTopic: 1
messageSize: 1024
payloadFile: "payload/payload-1Kb.data"
subscriptionsPerTopic: 1
producersPerTopic: 1
producerRate: 50000
consumerBacklogSizeGB: 0
testDurationMinutes: 15

As you can see from the configuration, this workload involves:

• one topic
• one partition, one producer, and one subscription on that single topic
• messages that are 1 kilobyte each (you can see the message payload here)
• clients producing 50,000 messages per second for 15 minutes
• no message backlog
   

But let's say that you wanted to change it up a bit and create a workload that involves:

• three topics
• five partitions per topic
• 5kb per message
• clients producing 500 messages per second for 30 minutes
   

You could specify that workload using this YAML file (called 3-topics-500-rate-5kb.yaml) and add it to the workloads folder in the benchmarks repo:

name: 3 topics / 5 partitions/topic / 5kb / 500 msg/sec
topics: 3
partitionsPerTopic: 5
messageSize: 5120
payloadFile: "payload/payload-5kb.data" # We would need to create this file
subscriptionsPerTopic: 1
producersPerTopic: 1
producerRate: 500
testDurationMinutes: 30

With your new workload configuration in place, you could deploy benchmarking infrastructure on Amazon Web Services (see instructions for Apache Kafka and Apache Pulsar), and then run your workload on Pulsar like this:

$ bin/benchmark \
  --drivers driver-pulsar/pulsar.yaml \
  workloads/3-topics-500-rate-5kb.yaml

If you'd set up Kafka infrastructure, you could run the workload like this:

$ bin/benchmark \
  --drivers driver-kafka/kafka.yaml \
  workloads/3-topics-500-rate-5kb.yaml

The results for your new workload would then be written to a file with a name like 3-topics-500-rate-5kb-Pulsar-2018-03-14-00-59-14.json. And that's it! You just created, ran, and obtained results for a custom messaging workload without touching a single line of code. Even better, that workload could be run on any supported messaging platform.

A Fresh Start for Messaging Benchmarks

We're confident that the OpenMessaging benchmarks project will emerge as the standard for the industry and eventually cover all major messaging platforms. Kafka and Pulsar are a good start (with RabbitMQ support coming soon), and we invite other communities to add support for other systems.

The new workload exercise we went through above shows just how easy it is to create new workloads. We invite you to experiment with your own workloads.

This post features contributions from Matteo Merli and Luc Perkins.

----------------------------------------------------
Thanks!
Matteo Merli