London to Paris Cycle for Prostate Cancer - Rider Analytics

With the London to Paris ride now complete, and the fundraising target well on its way, it’s the perfect time to dive into the technical magic that made it possible to track the journey in real-time. Over the past few days, you were able to follow the riders’ progress, see their locations on a map, and even view route details and elevation profiles through an intuitive dashboard.

Like the wizard of Oz (but a much more pleasant version), there was a very clever person behind the curtain. That person is Dean Cresswell, Solutions Engineer (extraordinaire). 

Dean has worked with many customers over many years and is one of the go to people for all things dashboards and mapping. It made perfect sense to entrust this important event tracking with him. So let's take a look at what Dean did to bring this all to life. Over to you Dean.

Data Collection: Gathering Location Data Across Multiple Devices

One of the main challenges we faced was tracking the location of all the riders in near real-time, using a variety of different devices. The team was equipped with hardware from several vendors, so we needed a solution that worked across platforms, was secure, and could deliver location data without delays.

After some research, we found the OwnTracks app, available for both iOS and Android, to be the perfect solution. Not only did it allow location tracking for multiple riders, but it also gave each rider control over when and how their location was shared. This was critical for maintaining data privacy. Additionally, OwnTracks could send location data in a JSON format via HTTP, which aligned perfectly with HTTP Event Collector (HEC), making data ingestion seamless.

Ingesting Data with Splunk’s HTTP Event Collector (HEC)

Once we had the location data from OwnTracks, we needed a reliable way to bring it into Splunk for analysis. The HEC allowed us to receive JSON payloads in near real-time and immediately start working with the data.

Initially, the system used the index time as the event timestamp, which wasn’t ideal due to the structure of the JSON body. Thankfully, Splunk’s flexibility in sourcetype configurations allowed us to quickly adjust this. We utilised Splunk's UI-based data onboarding tools to extract the correct timestamp, ensuring all location data was accurately timed.

With the data correctly ingested and timestamps aligned, we could begin the fun part. Turning this data into meaningful visualisations and analytics.

Turning Data into Insights

Once the location data was flowing into Splunk, our next challenge was presenting this information in a clear and engaging way for both the participants and those following the ride. We had a few key requirements for the dashboards:

1. Mapping the Ride

• We needed a map that could show:
• The planned route
• The completed portion of the route
• Each rider’s current location
• The elevation profile of the route, along with additional metadata such as each participant’s last reported location

To achieve this, we turned to the Maps+ app, developed by Scott Haskell and available on Splunkbase. This app provided advanced mapping capabilities that allowed us to display all the necessary details.

Splunk’s core features: schema on read and Search Processing Language (SPL). Both played a pivotal role here. These allowed us to manipulate and visualise the data without modifying the original dataset. We used SPL to:

• Plot the planned route.
• Separate completed portions of the route from the remaining distance, based on the latest location data of each rider.
• Apply the Haversine formula to calculate the distance from each rider’s current location to points along the route, snapping them to the nearest checkpoint.

We also used subsearches to overlay each rider’s actual location on the map, providing a real-time view of their progress.

2. Tracking Key Metrics

Mapping the ride was only part of the story. We also wanted to track key metrics that would give deeper insights into the riders’ performance and safety. These included:

• Battery percentage to monitor the impact of the app on riders’ devices.
• Connection status to ensure that each rider’s app was consistently sending location data.
• Lead rider identification to highlight who was in front at any given time.
• Safety monitoring, including alerts for riders who were falling behind the group.

We built these metrics into a dashboard using Splunk’s Dashboard Studio, which allowed us to create a visually engaging interface that resembled a virtual cycle computer. The single base search we used for the mapping features was easily expanded to include these metrics. Here’s how we tackled each one:

• Distance completed: We calculated the sum of the distance up to the furthest point reached by any rider.
• Lead rider: Using a simple eval comparison function, we determined which rider was closest to the furthest checkpoint.
• Elevation profile: We created an area chart showing the elevation data for the route, overlaid with a line indicating how much of that profile had been completed.
• Connection status: By comparing the last reported time for each rider, we flagged their status as "Good" if they had checked in within 30 minutes, "Fair" within an hour, and "Poor" if it had been longer.

We also used icons, shapes, and images to make the dashboard more visually appealing, ensuring it was as functional as it was engaging for viewers.

3. Making the Dashboard Public

You might be wondering, "How did you make this dashboard available for the public to view?" That’s where DashPub-Plus comes in. This open-source library allowed us to securely share the Splunk dashboard with the public, so anyone could follow the ride in real-time from anywhere in the world. DashPub gave us the flexibility to keep sensitive data secure while still making the dashboard accessible to everyone.

Conclusion

By leveraging Splunk’s powerful data ingestion, manipulation, and visualisation tools, we were able to track the riders in near real-time and present a clear, engaging view of their journey. Whether it was plotting the planned route, monitoring key metrics, or sharing the live dashboard with the public, each component came together to ensure a smooth and transparent tracking experience.

Thanks Dean, great work as always!

In the next blog, we’ll dive deeper into the technical details behind DashPub-Plus with a special guest, Will Searle, the current repo owner. We’ll explore how this library enabled us to share the dashboard with the world and ensure that everything worked seamlessly throughout the ride.

Once again, congratulations to the riders, and a huge thank you to everyone who followed along through our dashboards!