regreSSHion: Uncovering CVE-2024-6387 in OpenSSH - A Critical Vulnerability

OpenSSH, an application installed by default on nearly every Unix-like and Linux system, has recently come under scrutiny due to a critical vulnerability discovered by Qualys.

Designated as CVE-2024-6387 and aptly named "regreSSHion," this flaw exposes Linux environments to remote unauthenticated code execution. The implications of this vulnerability are far-reaching, potentially affecting countless servers and infrastructure components across the globe.

In this blog post, the Splunk Threat Research Team will dissect the technical intricacies of CVE-2024-6387, explore its potential impact on affected systems, and provide detection opportunities and mitigation strategies.

Key points about CVE-2024-6387:

• Vulnerability: Remote Unauthenticated Code Execution (CVE-2024-6387)
• Affected: OpenSSH 8.5p1 to 9.8p1
• Architecture: Primary PoC for 32-bit (x86), 64-bit (amd64) theoretically vulnerable
• Systems: glibc-based Linux distributions
• Impact: Root-level access if exploited
• Complexity: High (requires precise timing, multiple attempts)
• Status: Patch available (OpenSSH 9.8p1)

Technical Details

The Vulnerability

CVE-2024-6387 stems from a signal handler race condition in OpenSSH, affecting versions from 8.5p1 to 9.8p1 on glibc-based Linux systems. The flaw, a regression of an older vulnerability (CVE-2006-5051), allows remote attackers to execute arbitrary code as root, leading to full system compromise.

The nature of the signal handler race condition: The vulnerability occurs in OpenSSH's server (sshd) when handling the SIGALRM signal. In the affected versions, the SIGALRM handler calls functions that are not async-signal-safe, such as syslog(). This creates a race condition where the signal handler can interrupt critical sections of code, potentially leaving the system in an inconsistent state. Attackers can exploit this inconsistency to manipulate memory and execute arbitrary code.

Relation to the older CVE-2006-5051: Interestingly, this vulnerability is a regression of CVE-2006-5051, which was originally patched in 2006. The old vulnerability was also a signal handler race condition in OpenSSH. The fix implemented then inadvertently introduced a new vulnerability when changes were made to the logging infrastructure in OpenSSH 8.5p1 in October 2020.

This regression highlights the challenges of maintaining security in complex software systems over time.

Specific conditions for exploitation: Exploiting this vulnerability requires several conditions to be met:

• The target must be running a vulnerable version of OpenSSH (8.5p1 to 9.8p1) on a glibc-based Linux system.
• The attacker needs to time their exploit precisely to hit the race condition window.
• Multiple attempts are typically required due to the nature of race conditions.
• The exploit leverages specific behaviors of glibc's memory allocator, making it primarily effective on Linux systems.

The complexity of this exploit means that while the vulnerability is severe, successful attacks require a high level of skill and persistence. Nevertheless, the potential for remote code execution with root privileges makes this a critical issue for all affected systems.

Exploit Mechanics

The "regreSSHion" proof of concept exploit that was released on GitHub leverages a complex race condition, requiring precise timing and potentially thousands of attempts to succeed. Let's break down the key aspects:

Timing and Duration. The exploit's success hinges on hitting a very narrow time window. According to the Qualys research:

• On average, it takes about 10,000 attempts to win the race condition.
• With 100 connections (MaxStartups) accepted per 120 seconds (LoginGraceTime), it takes approximately 3-4 hours to win the race condition.
• Factoring in ASLR (Address Space Layout Randomization) bypassing, a full exploit could take 6-8 hours on average to obtain a remote root shell.

This timing is crucial because the exploit must interrupt specific operations at just the right moment to manipulate the system's memory state.

OS and Architecture. The PoC primarily targets 32-bit (x86) systems, though work on a 64-bit (amd64) version was in progress. Key points:

• The exploit was tested on Debian-based systems, specifically targeting glibc-based Linux distributions.
• The 32-bit version exploits weaknesses in ASLR implementation, making it easier to guess memory addresses.
• A 64-bit version would be more challenging due to stronger ASLR, but was considered feasible by the researchers.

Key Functions in the proof of concept. Let's look at two crucial functions in the proof of concept exploit.

a) prepare_heap():

This function is similar to setting up a complicated domino pattern. It arranges the computer's memory (the heap) in a very specific way, creating the perfect conditions for the exploit to work. Here's what it does:

• First, it creates 10 small memory chunks and immediately frees them. This is like placing 10 small dominoes at the start of our pattern.
• Then, it creates 27 pairs of memory chunks - one large (about 8KB) and one small (320 bytes). This is like setting up 27 pairs of large and small dominoes in a specific pattern.
• It fills these memory chunks with specific data. This is like marking our dominoes with special patterns that we'll recognize later.

The goal is to create a predictable layout in the computer's memory that the exploit can take advantage of later.

void prepare_heap(int sock) {
    // Packet a: Allocate and free tcache chunks
    for (int i = 0; i < 10; i++) {
        unsigned char tcache_chunk[64];
        memset(tcache_chunk, 'A', sizeof(tcache_chunk));
        send_packet(sock, 5, tcache_chunk, sizeof(tcache_chunk));
    }
   
    // Packet b: Create 27 pairs of large (~8KB) and small (320B) holes
    for (int i = 0; i < 27; i++) {
        unsigned char large_hole[8192];
        memset(large_hole, 'B', sizeof(large_hole));
        send_packet(sock, 5, large_hole, sizeof(large_hole));
 
        unsigned char small_hole[320];
        memset(small_hole, 'C', sizeof(small_hole));
        send_packet(sock, 5, small_hole, sizeof(small_hole));
    }
    // ... more heap manipulation ...
}

b) attempt_race_condition():

This function is where the actual exploit attempt happens. It's like trying to knock over our domino pattern at exactly the right moment to create a specific effect. Here's how it works:

• It prepares a specially crafted packet of data (like a perfectly weighted ball to knock over our dominoes).
• It sends all but the last byte of this packet to the server. This is like getting our ball ready to roll.
• Then, it waits for a very precise moment - just 1 millisecond before the server is expected to timeout the connection (SIGALRM). This timing is crucial.
• At that exact moment, it sends the final byte of the packet. This is like releasing our ball to knock over the dominoes at just the right time.
• Finally, it checks if the exploit was successful by looking at the server's response.

The exploit works by trying to interrupt the server's normal operation at a very specific moment. If timed correctly, it can manipulate the server's memory in a way that allows the attacker to run their own code with high-level (root) permissions.

int attempt_race_condition(int sock, double parsing_time, uint64_t glibc_base) {
    unsigned char final_packet[MAX_PACKET_SIZE];
    create_public_key_packet(final_packet, sizeof(final_packet), glibc_base);
 
    // Send all but the last byte
    if (send(sock, final_packet, sizeof(final_packet) - 1, 0) < 0) {
        perror("send final packet");
        return 0;
    }
 
    // Precise timing for last byte
    struct timespec start, current;
    clock_gettime(CLOCK_MONOTONIC, &start);
 
    while (1) {
        clock_gettime(CLOCK_MONOTONIC, &current);
        double elapsed = (current.tv_sec - start.tv_sec)
                         + (current.tv_nsec - start.tv_nsec) / 1e9;
        if (elapsed >= (LOGIN_GRACE_TIME - parsing_time - 0.001)) { // 1ms before SIGALRM
            if (send(sock, &final_packet[sizeof(final_packet) - 1], 1, 0) < 0) {
                perror("send last byte");
                return 0;
            }
            break;
        }
    }
    // ... check for successful exploitation ...
}

This process is extremely precise and requires many attempts to succeed. It's like trying to thread a needle while riding a rollercoaster — it might take thousands of tries to get it right. But if successful, it gives the attacker complete control over the server.

The complexity and precision required make this exploit challenging to pull off in real-world conditions. However, the potential impact if successful is severe, as it could allow an attacker to gain full control of a system without needing any login credentials. This is why it's crucial for system administrators to update their OpenSSH installations and implement other security measures to protect against such attacks.

What can defenders do?

The Splunk Threat Research team has developed a collection of Linux analytic stories that align with activity related to post-exploitation behaviors, providing defenders with powerful detection capabilities for Linux environments. Four key stories stand out in this collection:

• Linux Living Off The Land, which focuses on the abuse of native Linux binaries and utilities for malicious purposes
• Linux Privilege Escalation, covering techniques attackers use to elevate their privileges on a system
• Linux Persistence Techniques, addressing methods used to maintain long-term access to compromised systems
• Linux Rootkit, which tackles the challenging task of identifying sophisticated rootkits

These stories encompass a wide range of detection techniques, from anomaly detection to specific TTPs (Tactics, Techniques, and Procedures), all aligned with the MITRE ATT&CK framework. By implementing these analytic stories, organizations can significantly enhance their ability to detect, investigate, and respond to advanced threats targeting their Linux infrastructure across various stages of post-exploitation activity.

OpenSSH Inventory

To conduct an inventory search and verify if the OpenSSH version installed on an Ubuntu host is vulnerable to a specific exploit or attack, we can create a Splunk search query. This query will analyze package audit logs to identify the OpenSSH versions installed across our production network.

By leveraging the package audit logs, we can systematically track and audit all instances of OpenSSH installations. This approach ensures that we have a comprehensive overview of the software versions in use. Consequently, we can quickly identify any versions that are susceptible to known vulnerabilities.

Below is the simple search we created for this inventory:

index=unix source=package NAME= "*openssh*"
|  rex field=VERSION "^1:(?<ssh_version>\d+\.\d+)"
|  eval ssh_version_number = tonumber(ssh_version)
| eval vulnerable_ssh_version = if(ssh_version_number >= 8.5 AND ssh_version_number < 9.8, "Vulnerable SSH Version", "SSH Version not Vulnerable")
|  stats count by NAME VENDOR ssh_version ssh_version_number VERSION vulnerable_ssh_version

(Splunk query identifying OpenSSH packages, Splunk 2024)

Look for increased rates of “timeout before authentication” logs from sshd, on hosts running versions without the patch. Alternatively look for high levels of new connections if network monitoring is in place, as the attack requires repeated fresh login attempts.

sourcetype=journald OR sourcetype=linux:auth OR TERM(sshd) OR TERM(ssh)
TERM(Timeout) TERM(authentication) "Timeout before authentication"
| timechart count by host

(Splunk query identifying Timeout Before Authentication, Splunk 2024)

Mitigation Strategies

While patching is the most effective solution, there are several mitigation strategies organizations can employ to reduce their risk exposure to the CVE-2024-6387 vulnerability:

Patch Management

• Update to OpenSSH version 9.8p1 or later, which includes the fix for this vulnerability.
• If patching is not immediately possible, implement the following mitigations to limit exposure.

Access Control

• Limit SSH access to only necessary IP addresses and networks using firewall rules.
• Implement jump hosts or bastion servers to provide an additional layer of access control.

Host-based Intrusion Prevention

• Deploy host-based packages like fail2ban to identify and block potential exploitation attempts.
• Configure fail2ban to monitor SSH logs and automatically block IP addresses showing suspicious behavior.

SSH Configuration Hardening

Adjust the following settings in sshd_config to limit the effectiveness of attackers:

• LoginGraceTime:
  • Set LoginGraceTime to 0 to avoid the vulnerability.
  • Caution: This can potentially tie up system resources, as it removes the timeout for authentication attempts.
• MaxStartups:
  • Reduce MaxStartups to limit the number of unauthenticated connections.
  • Be aware that setting this too low may block legitimate users if an attacker is aggressive or if you have a high user count on the host.
  • Example: MaxStartups 10:30:100 (Start refusing connections at 10; refuse 30% of connections when there are 10-100 unauthenticated clients)
• PerSourceMaxStartups:
  • Set this to a small number to limit concurrent unauthenticated connections from a single source IP.
  • This is applied in addition to MaxStartups, using whichever limit is lower.
  • Example: PerSourceMaxStartups 5

Network Segmentation

• Implement strict network segmentation to limit the potential impact of a successful exploit.
• Use VLANs or network zones to isolate critical systems that require SSH access.

Multi-Factor Authentication (MFA): Implement MFA for SSH access as an additional layer of security. This doesn't prevent the initial exploit but can limit an attacker's ability to leverage compromised credentials.

Monitoring and Alerting

• Implement robust logging and monitoring for SSH services.
• Set up alerts for unusual SSH activity, such as high volumes of failed login attempts or connections with suspicious timing patterns.

Alternative Access Methods: For critical systems, consider implementing alternative secure remote access methods that don't rely on SSH, such as VPN solutions with strong authentication.

Snort: Network defenders can also leverage signature-based detection. Cisco Talos has released a Snort signature (SID: 63659) to detect exploitation attempts of CVE-2024-6387. This provides an additional layer of protection for organizations using Snort in their security infrastructure.

Remember, these mitigations should be part of a defense-in-depth strategy. No single measure is foolproof, and the most effective protection comes from combining multiple security layers. Regularly review and update your security measures to address new threats and vulnerabilities as they emerge.

Learn More

You can find the latest content about security analytic stories on GitHub and in the Splunk ES Content Update app. Splunk Security Essentials also has all these detections now available via push update. For a full list of security content, check out the release notes on Splunk Docs.

Feedback

Any feedback or requests? Feel free to put in an issue on GitHub and we’ll follow up. Alternatively, join us on the Slack channel #security-research. Follow these instructions If you need an invitation to our Splunk user groups on Slack.

Contributors

We would like to thank Michael Haag and Teoderick Contreras for authoring this post and the entire Splunk Threat Research Team for their contributions: Lou Stella, Bhavin Patel, Rod Soto, Eric McGinnis, Jose Hernandez and Patrick Bareiss. In addition James Hodgkinson of SURGe, and Jonathan Heckinger. Cisco Talos Michael Gentile and Keith Lehrschall.