A Case Study in Vulnerability Prioritization: Lessons Learned from Large-Scale Incidents

There’s no way around it: vulnerability management is complex. As organizations become more reliant on software and applications, the sheer volume of known vulnerabilities has become more difficult to track, prioritize, and remediate. Adversaries have also become increasingly reliant on exploiting vulnerabilities in order to compromise organizations. According to the Verizon 2024 Data Breach Investigations Report website, “14% of breaches involved the exploitation of vulnerabilities as an initial access step, almost triple the amount from last year’s report.”

Trends in vulnerability exploitation become more apparent when you consider large-scale incidents over the past few years, such as Log4j and MOVEit Transfer. This blog will provide an overview of the lessons learned from these incidents to help inform vulnerability prioritization.

Evaluating a Vulnerability

Before we dive into the case studies, here is a list of questions to consider when evaluating a vulnerability. These questions are grouped into five categories: who, what, when, where, and why. They are not listed in any order of importance and do not represent all possible considerations, but instead offer a starting point for vulnerability prioritization.

Who

• Peers: What is the level of social buzz surrounding this vulnerability?
  • Is any of this social buzz perceived as fear, uncertainty, and doubt (FUD) or fear-mongering?
• Industry: Is there guidance from trusted cybersecurity researchers or industry groups?
• Government: Are there alerts or emergency directives from government agencies?
• Customers: If you are a vendor, are customers inquiring about vulnerabilities?

What

• Is it remotely-exploitable? 
  • Can it be exploited over the network or does it require local access?
• Is proof-of-concept exploit code publicly available?
• What is the impact of successful exploitation? 
• Was this internally or externally discovered?
  • If discovered internally, is there a timeframe before public disclosure?
    • How long will it take to develop a fix?
    • Are customers impacted?
      • If so, how and when will you notify them?

When

• When was it disclosed?
  • Was it a coordinated/responsible disclosure with a deadline to patch before public disclosure?
• Is it actively exploited in the wild?
• Was it exploited as a zero-day? 

Where

• Is it present in your environment?
  • Is it a software component listed in a software bill of materials (SBOM), the Vulnerability Exploitability eXchange (VEX), or vendor advisory?
  • Is the impact of exploitation influenced by where the vulnerability is present in your environment?
  • Is it externally facing?
• Is a patch available?
  • If a patch is available, are you bound by a patch service-level agreement (SLA)?
    • Here is a post by Jacob Williams about rethinking patch SLAs. He points out that threat actors don’t care about your SLAs, and instead, it’s better to focus on how much risk you are willing to accept from delaying a patch.
  • What is the risk to your organization of leaving it unpatched?
  • Will patching this vulnerability lead to downtime?
  • Is there mitigation guidance that is preferred to a patch?
  • If a patch is not available, is there mitigation guidance as a workaround?

Why

• Does this vulnerability align with recent trends in adversary behavior?
• Does it have a score, such as CVSS or EPSS?
• Is it listed in a database like the CISA KEV? 

Case Study 1: Cybercrime and Zero-Day Vulnerabilities 

When news first spread in late May 2023 of a critical vulnerability in MOVEit Transfer, a popular file transfer solution, there were early signs pointing to the potential for mass exploitation. One cybercriminal group in particular, Cl0p, had been known to capitalize on vulnerabilities in file transfer solutions in order to rapidly target organizations. These attacks included Accellion File Transfer Appliance (FTA) in 2021 and GoAnywhere MFT in 2023. 

Cl0p’s spray-and-pray approach was part of a larger trend of ransomware groups moving away from encryption-based attacks and instead relying only on data exfiltration and extortion to reach more victims. Adding to the urgency, the cybersecurity firm Mandiant confirmed to Bleeping Computer that the vulnerability was a zero-day with exploitation observed as early as May 27th, days before it was publicly disclosed. The MOVEit breach ultimately impacted more than 2,700 organizations and nearly 95.8 million people, according to the cybersecurity firm Emsisoft. 

There are three key lessons in vulnerability prioritization from the MOVEit breach: 

1. Adversary behavior can inform our expectations about the likelihood of exploitation.
In order to understand the urgency surrounding a newly disclosed vulnerability, it’s important to have broader context, including an awareness of cybersecurity trends. A critical vulnerability in a file transfer appliance carried additional urgency in 2023 because of the strategies employed by cybercriminals. 

This is backed up by SURGe research that found MITRE ATT&CK technique T1190 (Exploit Public-Facing Application) was the highest reported initial access technique when evaluating public and private cyber incident reports from 2023. Red Canary observed in their 2024 Threat Detection Report that exploitation of critical vulnerabilities in 2023 often involved an attack path beginning with exploitation of a public-facing server or web application.

2. Simple attack chain + zero-day = higher priority 
The MOVEit vulnerability was first exploited as a zero-day in attacks that did not require lateral movement beyond the MOVEit application. Researchers at Huntress reported, “the attack vector offers the ability to detonate ransomware right away,” which allowed Cl0p to exfiltrate data directly from the application. The ease of exploitation and lack of public awareness as a zero-day made the MOVEit vulnerability a high priority for organizations once it was publicly disclosed. 

One caveat here is that not all zero-day vulnerabilities are a high priority; there are other factors to consider, such as attack complexity. Katie Nickels, Director of Intelligence Operations for Red Canary, explained this well in a talk at ShmooCon 2024 titled, “Why We Need to Stop Panicking about Zero-Days.”

3. Supply chain considerations
Vulnerabilities in popular applications carry a higher risk of supply chain compromise. Some organizations were impacted by the MOVEit breach because their vendor’s contractor’s subcontractor used MOVEit Transfer, Emsisoft reported. This risk highlights the importance of vendor risk assessments and other key practices outlined in NIST’s Cybersecurity Supply Chain Risk Management (C-SCRM) guidance.

Case Study 2: The “Unknown Unknowns” in Your Environment

In December 2021, SURGe published a blog with early guidance on how to detect exploitation of a critical vulnerability in the popular open source Apache Log4j logging library. Several factors made this vulnerability a high priority for organizations:

• It was remotely exploitable.
• It was publicly disclosed before a fix was available.
• Proof-of-concept exploit code circulated on the internet.
• Because Log4j was popular among Java developers, it was embedded into thousands of software packages and integrated into millions of systems worldwide.

The US Cyber Safety Review Board (CSRB) assessed in their review of the incident: “Log4j is an ‘endemic vulnerability’ … vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer. Significant risk remains.” 

Organizations struggled to quickly identify where Log4j was present in their environment – often one of the first steps in vulnerability prioritization – as open source software components aren’t always readily listed. The CSRB observed, “few organizations were able to execute this kind of response, or the speed required during this incident, causing delays in both their assessment of the risk and in their management of it.” These vulnerable “unknown unknowns” highlight the need for organizations to maintain an accurate asset and application inventory. 

Months before the Log4j vulnerability was disclosed, the Biden Administration issued an executive order in May 2021, which mandated a report on the minimum elements for an SBOM. An SBOM is essentially an ingredients list of the components used in building software, along with their supply chain relationships. Widespread SBOM adoption can go a long way in improving how organizations identify and respond to future vulnerabilities in open source software. 

Case Study 3: Disruption from a Breach vs. Downtime

Responding to a vulnerability isn’t always as simple as “just patch.” Organizations often need to consider the risk to their organization and the impact of system downtime during the patching process. An example of this can be found in a 2021 cyber espionage campaign targeting on-premises Microsoft Exchange servers. Microsoft attributed the attacks to HAFNIUM, a state-sponsored threat group linked to China. The campaign leveraged multiple zero-day vulnerabilities, which were soon exploited by malicious actors outside of HAFNIUM.

In the months following this announcement, Microsoft released security updates for Exchange along with an on-premises mitigation tool for customers before a full patch was available. Reporter Andy Greenberg outlined the many challenges associated with patching on-premises Exchange servers in an article for WIRED titled “Your Microsoft Exchange Server Is a Security Liability.” The patching complexity was due in part to the age of the code base for on-premises Exchange and the software interdependencies that could prevent an update from completing.

Not only was it difficult for organizations to patch these vulnerabilities in a timely fashion, but they also needed to hunt for potential backdoors left behind by the adversary. A little more than a month after Microsoft disclosed the HAFNIUM campaign, federal officials announced a court-authorized operation where the FBI removed malicious web shells from hundreds of victims’ systems in the United States. SURGe released blogs with guidance for detecting exploitation in on-premises Exchange servers and cleaning up the webshells. 

As an industry, we need to do a better job of building code that is secure-by-design. But, we also need customers to do a better job of knowing what technology they use, applying patches and recommended secure configurations, and effectively planning to manage end-of-life for hardware and software beyond their vendor-supported lifespans.

Some services currently rendered from on-premises boxes would be more securely rendered from the cloud. Some of the reluctance about moving to the cloud may stem from the perception that the lift and shift has initial costs and ongoing operating expense implications. But, relying on aging infrastructure that is too old to patch as the core of your business? That unmanaged risk has costs, too. They just don't show up on your income statement or balance sheet.

Vulnerability Databases and Scoring Methods

In addition to the use cases above, it’s worthwhile to include an overview of the current databases and scoring frameworks to evaluate emerging vulnerabilities, along with their strengths and limitations.

• CISA’s Known Exploited Vulnerabilities (KEV) catalog is considered an authoritative source for critical vulnerabilities that have been exploited in the wild. In order to make it in the KEV, vulnerabilities must have an assigned Common Vulnerabilities and Exposures (CVE) identifier, evidence of exploitation, and a pathway for remediation. The Verizon 2024 Data Breach Investigations Report found, “By doing a survival analysis of vulnerability management data and focusing on the vulnerabilities in the CISA Known Exploited Vulnerabilities (KEV) catalog, (arguably an area of priority focus in vulnerability management)... it takes around 55 days to remediate 50% of those critical vulnerabilities once their patches are available.”
  
  SURGe analyzed KEV data as part of our 2023 Macro-level ATT&CK analysis, and we found that the average time between when a vulnerability was disclosed that year and when it was known to be actively exploited as a KEV entry was seven days. Interestingly, our analysis showed that the higher CVSS scores weren’t likely to be exploited any faster than the lower scores. Our data also showed that most KEV entries have a CVSS score above 4.0.
  
  
  ^{Days between CVE release and Known Exploit vs. CVSS Score}
  
  While the KEV is important to include in an organization’s vulnerability management strategy, it isn’t the full picture. Cisco and Cyentia Institute’s 2023 Prioritization to Prediction report found that 94% of CVEs that have exploitation activity aren’t included in the KEV catalog.  
  
  
• National Vulnerability Database (NVD) is the U.S. government’s repository of vulnerabilities that have been assigned a CVE identifier. Vulnerability severity is measured using the Common Vulnerability Scoring System (CVSS), which takes into account a base score plus temporal and environmental metrics. However, a 2022 report from Cyentia Institute, now part of Cisco, found that prioritizing vulnerabilities based solely on the existence of exploit code is 11 times more effective than CVSS in minimizing exploitability. The report also found that evaluating X (formerly known as Twitter) mentions to prioritize software fixes is twice as effective as using CVSS. While the NVD provides the most comprehensive list of CVEs anywhere, the site recently dealt with a backlog in CVE analysis and scoring. Cisco Talos reported in April 2024 that the NVD was analyzing about 2.9% of all published CVEs it received. In May 2024, NIST announced it would receive additional processing support for incoming CVEs, helping to return the NVD to past processing rates within a few months. 

• CVE.ICU features a collection of CVE-related data that is extracted from the NVD by an automated system. The source code is publicly available to encourage collaboration and a better understanding of the vulnerability landscape, according to lead researcher Jerry Gamblin. 

• Exploit Prediction Scoring System (EPSS) was created in 2019 and uses machine learning to predict the probability that CVEs will be exploited within the next 30 days. EPSS is governed by the Forum of Incident Response and Security Teams (FIRST) and relies on the existence of an assigned CVE ID, which can make it less useful to software suppliers, computer security incident response teams (CSIRTs) and bug bounty programs. You can read more about the methodology behind EPSS here. 

• Stakeholder-Specific Vulnerability Categorization (SSVC) system is an analysis methodology created in 2019 by Carnegie Mellon University's Software Engineering Institute (SEI), in collaboration with CISA, to account for “a vulnerability’s exploitation status, impacts to safety, and the prevalence of the affected product in a singular system.” CISA uses its own SSVC decision tree model to prioritize vulnerabilities.

• Common Security Advisory Framework (CSAF) is an OASIS standard and language used to exchange security advisories. According to OASIS, CSAF “plays a crucial role in the cybersecurity arena since it allows stakeholders to automate the creation and consumption of security vulnerability information and remediation.”

• Vulnerability Exploitability eXchange (VEX) provides information about the status of a vulnerability in specific products, such as whether or not the product is affected, fixed, or under investigation. The VEX format is machine-readable and developed as part of the National Telecommunications and Information Administration (NTIA) Multistakeholder Process for Software Component Transparency. 

• Cisco VM (formerly Kenna.VM) is a commercial solution that helps to centralize vulnerability and asset information in order to reduce noise so that vulnerability management engineers can focus on the most critical vulnerabilities and streamline remediation efforts. 

• The OSV Schema is a data format that was created to map to open source versioning schemes. The OSV schema can be used by both open source consumers and vulnerability databases to aggregate and identify vulnerability data. 

• The OSS Index is a catalog of open source components and scanning tools maintained by Sonatype to help developers identify vulnerabilities and better understand risk. 

• VulDB (formerly VulnDB) is a database provided by pyxyp inc. that includes more than 272,000 vulnerability entries, including threat intelligence information. 

• Vulnerability Impact Scoring System (VISS) is an open, customizable framework to evaluate the impact of security vulnerabilities from a defender’s perspective. VISS is owned and managed by Zoom.
  

Conclusion

The above case studies demonstrate how vulnerability prioritization can be influenced by factors such as adversary behavior, supply chain risk, open source software components, and the effects of downtime. These takeaways do not encompass all of the challenges associated with vulnerability prioritization, but they are a starting point. Organizations should view vulnerability prioritization as an iterative process of continual improvement influenced by an evolving threat landscape. As organizations mature and refine their vulnerability management program, they can begin to implement additional steps such as continuous monitoring, automation, and the integration of vulnerability management tools with configuration management databases (CMDBs) to improve detection and remediation.

In the future, we may see software suppliers adopt a secure-by-design philosophy that takes greater ownership of customer security outcomes. This could be influenced by future regulations, heightened liability for security executives, and the loss of customer trust. We may also see new use cases for emerging technologies, like machine learning and artificial intelligence, to help speed up and guide the vulnerability prioritization process, while maintaining a human-in-the-loop approach. In the meantime, we can anticipate a growing number of emerging vulnerabilities, emphasizing the need for a vulnerability management program that accounts for the nuance and complexity required to create an effective prioritization strategy.