Regeneron Harnesses Science and Splunk to Develop Life-Changing Medicines

Key Challenges

Regeneron scientists required flexible, reliable, and real-time monitoring to provide visibility into experiments, equipment status, and quality control processes.

Key Results

Powered by Splunk Cloud Platform, Regeneron created holistic dashboards of critical data in lab environments to enable the company to increase the development speed of life-changing therapies.

Download PDF

When developing life-changing medicine, every second counts.

From slowing down vision loss to treating cancer, Regeneron uses the power of science to invent and develop transformative new medicines for patients in need. Therefore, it’s critical to empower the company’s scientists and engineers with real-time insights into the status of lab equipment and experiments. Earlier monitoring systems only allowed scientists to view the status of experiments from a PC directly connected to the lab equipment. Meaning, scientists couldn’t multitask, significantly slowing down experiments and Regeneron’s ability to produce life-saving therapies. In addition, Regeneron quality control (QC) scientists had no visibility into issues or delays in the sample analyzing process, resulting in discarded samples, causing further delays.

Regeneron turned to Splunk to help create a reliable, real-time monitoring solution for lab data that scientists could access and gain full visibility into QC bottlenecks. Now, experiments aren’t hampered by scientists’ inability to physically transition from one room to another, and samples are tested in priority order, resulting in increased speed of medicine production for treatment delivery to those who need it most.

Outcomes

significant data loss events

800-hour

reduction in instrument downtime

Near-perfect

patch compliance

Critical lab data — all at a scientist’s fingertips

The solution? A custom dashboard showing Regeneron’s equipment status — “Health TV” as the team calls it — powered by Splunk Cloud Platform. Here’s how it works: Universal forwarders are deployed to computers attached to scientific equipment. These universal forwarders pick up the log files from the equipment and send them to Splunk Cloud where they are stored at the indexer layer. Then, Regeneron runs multiple searches against it in near-real time to produce insight into the current state of equipment based on its filtering criteria and common language model, which then gets presented as a map with overlays of scientific equipment status.

“Health TV” provides Regeneron scientists with a holistic view of the status of their labs and equipment from anywhere, including the comfort of their offices where they can take calls, respond to emails, and perform other meaningful work while keeping a watchful eye on the experiments. In fact, since implementing “Health TV,” there have been no significant data loss events resulting from PC-based instability that would require scientist or IT intervention. Through “Health TV,” Regeneron scientists have peace of mind knowing that experiments are running smoothly and can see exactly when time-sensitive tests are complete.

“From microscopes to slides with injection pipettes, all this data is ingested into Splunk,” says Keith Keimig, associate director, cybersecurity monitoring at Regeneron. “In ‘Health TV,’ we built a common language for the equipment status and error state and then visually present that on a map of the lab so our scientists can easily keep watch on pertinent information, with specialized alerts coming through text, email and on ‘Health TV’.” Optimizing a scientist’s time in this way may seem like a small change, but knowing exactly when an experiment has completed, or even errored out, without the need to stand right in front of it, reduces the overall experiment time, allowing for even more experiments to be run. “Our scientists are able to get more work done so they can get new medicines to the market and find new medicines sooner,” says Keimig.

Our scientists are able to get more work done so they can get new medicines to the market and find new medicines sooner.

Keith Keimig, Associate Director, Cybersecurity Monitoring, Regeneron

For Regeneron, preventing equipment downtime changes lives

Because experiments are now run more efficiently, “Health TV” also increases equipment usage, helping Regeneron get more value from its investments. With the insights captured in “Health TV” and faster error identification, instrument downtime in the genotyping lab has been reduced by 800 hours. “We had 35 ‘blue screens of death’ in the last year on computers attached to scientific equipment that got corrected with driver updates that likely would never have been reported prior to ‘Health TV,’” says Keimig. Security has benefits as well: The labs where “Health TV” is deployed now have near-perfect patch compliance.

“Health TV” also helps Regeneron prevent critical lab equipment downtime by alerting about scheduled maintenance well before any alerts go off. “Our risk for downtime is reduced because we’re able to track and address issues preemptively,” says Keimig. “For instance,” continues Keimig, “several of our hard drives that are written to are mirrored disks. One side would have errors and fail, but the other side would keep running. Previously, there was no notification when this occurred.” As part of this project, the IT team implemented a ticketing system so that when they see errors on disk drives, technicians now come in and replace them, saving the company not only a significant amount of money from costly outages, but allowing them to create medicines faster and more efficiently — changing lives in the process.

Bringing science (and data) to life

“Health TV” is also used by scientists in the Regeneron QC manufacturing facility, where samples are evaluated to ensure they’re within the strict threshold of acceptability. Previously, there was no visibility into issues or delays in sampling and analyzing quality before being released as acceptable. Leveraging “Health TV” via Splunk Cloud Platform, the team created a real-time view of the QC hold times, displaying where critical delays are starting to occur.

“In the manufacturing facility, ‘Health TV’ is displayed on centralized screens in the lab and on the plant floor as an easy-to-read stoplight dashboard exhibiting how many samples are in the queue, how many are coming in, and the status of those samples against their SLAs,” says Keimig. “Some samples only have a 24-hour lifespan. So, if the sample isn’t evaluated within that time frame, it must be thrown out.”

And losing even one sample in this manner has a significant impact on overall operations. “It slows down the whole process by 24 hours, or until the next sample batch is ready,” says Keimig. “Prioritized testing and assessment of samples in real time is a game-changer.” The data provided by Splunk Cloud Platform and displayed in “Health TV” provides quick links to lab technicians to assess the wait times of samples for issue identification, resolution, or proper discarding.

Being able to run sample evaluation tests faster and more efficiently allows Regeneron to bring more treatments to cancer patients and helps the company better fight cardiovascular diseases and rare and infectious diseases.

The future of “Health TV”

“When I joined Regeneron, I took a challenge from my CISO to ‘bring Splunk to the scientists’. Five years on, the Research IT and IOPS IT teams are finding potential use cases every day for Splunk as we continue to uncover what hurdles our scientists’ have,” says Keimig. These future use cases include leveraging AI to predict freezer status based on temperature fluctuations and humidity, as well as the next big launch to support the business: a mobile version of “Health TV” so scientists can gain further access to critical experiment statuses.

When I joined Regeneron, I took a challenge from my CISO to ‘bring Splunk to the scientists’. Five years on, the Research IT and IOPS IT teams are finding potential use cases every day for Splunk as we continue to uncover what hurdles our scientists’ have.

Keith Keimig, Associate Director, Cybersecurity Monitoring, Regeneron