Nothing PUNY About OpenSSL (CVE-2022-3602)

Authors and Contributors: As always, security at Splunk is a family business. Credit to authors and collaborators: David J. Bianco, Ryan Fetterman, Michael Haag, Tony Iacobelli, Ryan Kovar, Marcus LaFerrera, Audra Streetman, Casey Wopat 

For details on how to find CVE-2022-3602, skip down to the “detections” sections. Otherwise, read on for a quick breakdown of what happened and ideas on how to detect this vulnerability.

Introduction to CVE-2022-3602 and CVE-2022-3786

On October 25th, 2022 the OpenSSL project team announced that a forthcoming release of OpenSSL would contain a security fix with a severity of critical. This naturally piqued our interest, especially since the last time there was a critical severity in OpenSSL was all the way back in 2016 which could potentially lead to execution of arbitrary code. Thankfully, that vulnerability was limited in scope, albeit noteworthy.

So, we waited patiently for more details. Then on November 1, 2022, OpenSSL released a security advisory with a severity of HIGH, disclosing two buffer overrun vulnerabilities that can be exploited to cause denial-of-service, and one of them potentially permitting remote code execution. These vulnerabilities are formally tracked as CVE-2022-3786 and CVE-2022-3602. In this blog post we will be mostly focusing on CVE-2022-3602. 

CVE-2022-3602 (or as it is increasingly known, SpookySSL) is triggered by sending an X.509 certificate with a specially crafted email address to a vulnerable client or server. This vulnerability is limited to OpenSSL v3.0 to v3.0.6. Thankfully the kind developers behind OpenSSL released a patch along with the security advisory. So, if you are affected, follow the age-old wisdom and patch, patch, patch. As of the writing of this blog, the biggest impact is the potential for denial of service attacks. We are not aware of any remote code execution (RCE) exploits related to this vulnerability. However, with all vulnerabilities of this class, that is possible. 

Now we’re not gonna lie, this is a technical bug with a week of breathless anticipation around it. In the section below, we outline our interpretation of the vulnerability and what you can do to detect it in your environment. But before we go in depth, here are the key things you need to know: 

• The exploit is triggered by an invalid email address using crafted punycode in the Subject Alternative Name (SAN) extension of a X.509 certificate.
• This will allow for either denial-of-service or possible remote code execution (RCE).
• To detect exploit attempts you will need to collect X.509 certificates using a tool like Zeek.
• Make sure you check for more than just the operating system’s version of OpenSSL. Applications might have the vulnerable OpenSSL bundled.
• CISA recommends people keep an eye on the Dutch Nationaal Cyber Security Centrum (NCSC-NL) github repo for updates and a holistic industry view.

We go into much more detail below about how the vulnerability works and some thoughts on possible exploitation methods. If you want, feel free to skip down to the hunting queries.

E-mail is PUNY

Before we can talk about how this vulnerability works, we need to understand X.509 certificates, specifically how X.509 handles internationalized (i.e., non-ASCII) email addresses. Buckle up.

Subjects and Names

Every TLS certificate has a subject, which is the name of the entity to which that certificate is assigned. For example, a website certificate might have a subject name corresponding to its Internet hostname, such as www.splunk.com. For X.509 certificates, we use a subject’s distinguished name, which looks like /C=US/ST=California/L=San Francisco/O=Splunk LLC/CN=www.splunk.com. By presenting this certificate to the client, a web server can prove its identity as www.splunk.com when the browser validates the certificate’s digital signatures. 

If the same certificate applies to multiple entities (e.g., a server that responds to both splunk.com and www.splunk.com), the X.509 Subject Alternative Name (AKA subjectAltName or SAN) extension can be used to name additional subjects. A certificate must have a primary subject, but it may have as many alternate names as the owner likes (including zero alternate names). Alternate names are very common. 

For example, a certificate for the main website at www.splunk.com might also be used for the hypothetical (for now!) site surge.splunk.com. In this case, the primary subject would probably be /C=US/ST=California/L=San Francisco/O=Splunk LLC/CN=www.splunk.com while one of the alternate names might be /C=US/ST=California/L=San Francisco/O=Splunk LLC/CN=surge.splunk.com.

Email Addresses in Certificates

Optionally, certificates can embed email addresses into their subject names (either the primary subject or any of the alternate names). The name can explicitly be an email address type (e.g., email: skornbloom@splunk.com) or the email might be one attribute in a longer distinguished name, such as /C=US/O=Shelly Kornbloom/U=Shelly Kornbloom/emailAddress=skornbloom@splunk.com. A common use for these types of subjects is for certificate-based client authentication, where a browser might present a certificate issued to the user in order to authenticate to the application on the other side. Once the server successfully validates the client’s certificate, it can rely on the fact that the user is who they say they are, in this case, Shelly Kornbloom. 

Non-ASCII Characters in Email Addresses

So far, all those examples use the English alphabet exclusively, which works well for some people, but not all. For domains with international characters (unicode) or even emoji (yeah, 🤯, right?) we need to use an encoding called punycode to translate them to plain ASCII. If Shelly were living and working in Japan instead of the US, his personal certificate might have punycode in the email address, looking something like /C=JP/O=Shelly Kornbloom/U=Shelly Kornbloom/emailAddress=skornbloom@xn--pckta4g2c5b.com. That xn–pckta4g2c5b portion of the domain name is the punycode, which when fully decoded represents the スプランク.com domain, as you can see in this example:

Punycode translates a unicode domain of an email address into ASCII, but only the domain. Punycode cannot be used for the mailbox name (skornbloom in this example). But what if we have another Japanese Splunker named Haruto (陽翔)? Fully internationalized, his email address might be 陽翔@xn--pckta4g2c5b.com. This is often called the SMTPUTF8Mailbox format, since the mailbox name (the username portion of the address) is now written with UTF-8 characters. Inside a certificate, this would appear as a name that looked something like othername: SmtpUTF8Mailbox::陽翔@xn--pckta4g2c5b.com.

X509 Name Constraints

No, sorry, we’re still not quite ready to talk about the vulnerability. Next we have to talk about how Certificate Authorities (CAs) delegate authority to other CAs. 

The whole job of any CA is to sign certificates. There’s a hierarchy of CAs, starting at the top with a trusted Root CA, and ending with whichever CA signed the server or user entity’s certificate (this is called the Issuing CA). There may also be one or more CAs between the root and issuer (Intermediate CAs). Any certificate for a host or user will also contain full copies of all these other CA certificates. It’s called the certificate chain. Certificate recipients use it to verify each of the signatures in the chain during certificate validation.

Delegating CA Authority

Any CA can delegate all or part of its authority to the next CA down the chain. Consider a large organization that has its own root CA because it issues its own internal certificates. Maybe Splunk has a CA it uses for Canadian certificates, and another for US certificates. The CA CA (haha) is only allowed to issue certificates for Canadian entities. Likewise, the US CA can only issue certificates for US entities. But both certificates are signed by Splunk’s custom root CA. How can we make sure the CA and US authorities follow these rules?

The Name Constraints Extension

That’s where the X.509 Name Constraints extension comes in. It’s an optional feature of a CA certificate. If present, it means the signer of that CA certificate is delegating only part of its power, and in order for an issued certificate to be valid, it must conform to the specified constraints. Constraints can either permit (an allow list) or exclude (a block list) certain values in subject names.

Name constraints can be placed on a number of different types of data: DNS domains and hostnames, IP addresses, portions of the subject namespace (known as directories), and most importantly for this discussion, email addresses. Name constraints don’t actually prevent a CA from going rogue and issuing whatever certificates it wants, they just prevent certificates that fall outside of those boundaries from validating properly, rendering them useless.

Constraints on Non-ASCII Email Addresses

Placing an email address name constraint on a CA certificate means that any email addresses in the subject or alternate names in the issued certificates must obey the constraint. Constraints on email addresses can include domains, subdomains, and hostnames used in email addresses, or even name a specific mailbox. They may be plain ASCII throughout, or they may include punycode and/or SMTPUTF8Mailbox components. For example, a “permitted” constraint of email:splunk.com means that IF, AND ONLY IF, the issued certificate contains an email address in the subject or any of the alternate names, that address must be in the @splunk.com domain, but that the exact mailbox name is irrelevant. On the other hand, a “permitted” constraint of othername: SmtpUTF8Mailbox::陽翔@xn--pckta4g2c5b.com means that, if present, the address must be EXACTLY othername: SmtpUTF8Mailbox::陽翔@xn--pckta4g2c5b.com (an exact match including the mailbox name) and no other values are allowed. 

When comparing a subject’s SMTPUTF8Mailbox format address to an email address constraint, any punycode portions of the domain must be decoded back into their original Unicode characters (behavior is specified by RFC8398). Note that this is only necessary for SMTPUTF8Mailbox addresses. Standard email address formats are not decoded this way, even if they contain punycode in the domain.

Putting It All Together: How the Vulnerabilities Work

This is where the trouble begins. In OpenSSL versions 3.0.0 - 3.0.6, there were two methods to trigger a vulnerability in the punycode decoder. By crafting a certificate with a chosen SmtpUTF8Mailbox address in the alternate names, an attacker can insert malicious punycode which, when decoded during validation, would result in overflowing the stack with 4 bytes of the attacker’s choosing (CVE-2022-3602) or by an arbitrary number of ‘.’ characters (decimal 46) (CVE-2022-3786).

From what we currently know, the following conditions must be met in the crafted certificate:

1. An issuing CA must have signed a malicious certificate. The certificate must have a valid signature chain, all the way up to a trusted root CA. Failing this, the application must be configured to continue with certificate validation even when the signature chain can’t trace back to a trusted CA.
2. At least one of the intermediate CA certificates must have an email address constraint, which can be either a “permit” or an “exclude” constraint.
3. The malicious certificate must include the Subject Alternative Name extension, and must have an SmtpUTF8Mailbox format email address in at least one alternate name.
4. That SmtpUTF8Mailbox format email must have at least one portion of its domain encoded as punycode.
5. The encoded punycode must be specially chosen by the attacker to trigger the decode vulnerability and overflow the stack.

Note that while the email address to be validated must contain UTF-8 characters in the mailbox name, it’s really the punycode in the domain name that is the malicious part. The whole name must be a SmtpUTF8Mailbox format name, though, because that’s the only circumstance in which the specification requires the punycode to be decoded in order to compare the address with the constraint. Even if you have an email address with malicious punycode but no mailbox name, or the constraint’s mailbox name is pure ASCII, the punycode will never be decoded and the vulnerability will not be triggered.

Server-side Vulnerability

The official OpenSSL security advisory states that the server is vulnerable when it tries to validate certificates from malicious clients, but only if the server is using client certificate authentication. In this scenario, the TLS negotiation will explicitly request a certificate from the client. Without client certificate authentication, there’s no opportunity for the client to present a malicious certificate.

Additionally, the vulnerability requires the malicious certificate to have either a valid signature chain all the way to the root CA, or for the server to be configured to continue validation despite an invalid signature chain. This is not the default configuration for most services.

Client-side Vulnerability

A TLS client would be vulnerable if it connects to a malicious server. 

Additionally, the vulnerability requires the malicious certificate to have either a valid signature chain all the way to the root CA, or for the server to be configured to continue validation despite an invalid signature chain. It is somewhat more common for TLS clients to ignore signature verification, but is unlikely to be the default for most applications.

Exploit Effects

At a minimum, successful exploit attempts will cause a server or client to crash. As the advisory states, stack overflow protection may be able to keep an attacker from executing arbitrary code, though execution remains a possibility, especially for those systems lacking overflow protection.

Detecting Something PUNY…

… is going to be tricky. There are a few issues here that combine to make this vulnerability difficult to detect. Fortunately, they can be overcome, but it might be a little while before reliable detection is available.

Know Thyself, but Knowing Your OS Version May Not Be Sufficient

Even before the OpenSSL team dropped the details of the vulnerability, there were suggestions that you could determine your potential exposure by using the openssl version command from the command line. The problem with this is that while it identifies the default version of OpenSSL installed on your system, it may not accurately identify the version that your running services are actually using. 

It is common for programs to be released bundled with their own versions of OpenSSL, and thus it is possible for the system default version to be vulnerable, but the version you’re running on a given service to be safe, and vice versa. So use the output of the openssl version command as a hint if you find it useful, but don’t solely rely on it to determine exposure. Marcus Hutchins’ MalwareTech blog has a great article on how to find out what version of OpenSSL a service actually uses. 

Visibility Issues

In order for your IDS or your SIEM to be able to detect exploits for this vulnerability, they must first be able to see the subject alternate names in all your X.509 certificates. Many systems will find this challenging, probably because they’ve never had to examine those parts of the certificate before and thus their parsers may not yet be decoding the alternative names which are, after all, technically an optional extension. 

It may be worthwhile to check with your network sensor provider to see if their X.509 decoder 1) exists, and 2) is capable of seeing and logging the alternative names.

Punycode Detection

Lastly, even if you do have access to the alternative names, exploiting this vulnerability will still require the attacker to provide malformed punycode. Right now, it’s unclear exactly what this would look like, or even if you could distinguish well-formed punycode from malicious punycode without actually trying to decode it. 

Most IDS and SIEM solutions will struggle with decoding, since they are more suited for matching known values or correlations between related events than they are for performing computations to identify malicious input. Splunk’s CyberChef app might be useful here, however we have not yet been able to test this with a real world malicious payload.

For Zeek data we know some subject alternative names are parsed out, and then there is a “other_fields” field. We can use a CyberChef recipe, but we may need to add some regex. 

First we will extract any xn– domains using regex against the san.email and san.other_fields. 

index=zeek sourcetype="zeek:json"
| rex field=san.email{} "\@(?<domain_detected>xn--.*)"
| rex field=san.other_fields{} "\@(?<domain_detected>xn--.*)"
| table domain_detected

We see some of the punycode domains.

Let’s now add our CyberChef recipe. This recipe uses the jsonrecipe to convert the domain_detected field from punycode to ASCII. During our testing we found that if a field has multiple values, CyberChef will not decode it properly.

index=zeek sourcetype="zeek:json"
| rex field=san.email{} "\@(?<domain_detected>xn--.*)"
| rex field=san.other_fields{} "\@(?<domain_detected>xn--.*)"
| cyberchef infield='domain_detected' outfield='convertedPuny' jsonrecipe="[{"op":"From Punycode","args":[false]}]"
| table domain_detected convertedPuny

In addition, we tested against a subdomain punycode xn--pckta4g2c5b.xn--pckwbd2b8byg0c.com and the output is accurate, but it does not transform correctly and looks like random characters. If items do not decode properly, the next step may be to decode it directly on CyberChef from punycode. Within the jsonrecipe there is an argument field that defaults to false. Switching this to true should convert the domain name to true language. 

The presence of punycode in the email fields within X.509 based logs does not indicate the presence of evil. As potential adversaries start to kick off scans to identify vulnerable hosts, expect to see every email field fuzzed in hopes something occurs. It may be easier to find any `xn--` domains in use and convert them manually in CyberChef to identify suspect events. To provide context while hunting, source by ingress to web servers or edge devices that are exposed publicly to the Internet. Client to server (egress) may have limited instances due to browsers not using OpenSSL by default. Potentially the best datasource will be an IDS or edge device that has signatures for this vulnerability.

Splunk Enterprise Security and ESCU

While we have spent some time explaining this attack and we realize that additional effort needs to be put toward investigating this, don’t forget  the basics are still important. Asset management, hopefully via the asset and identity framework in Enterprise Security (ES), will tell you where your vulnerable systems reside. Running regular vulnerability scans that integrate into Splunk will display which systems are vulnerable and can help you prioritize your patching schedule and better focus your detection efforts. For folks using Enterprise Security Content Updates (ESCU), our Security Research team will release a new Splunk Analytic Story as soon as possible, containing detections for this threat. 

Splunk Services

Our team of Security Professionals, that are part of our Splunk Professional Services team, can help you to implement everything we’ve mentioned here. We also have more targeted offerings which can help you increase your security posture as well. 

Conclusion

We’ve been through a lot in this blog post. We think we’re pretty good friends now. So, as a friend, the best advice we can give is to patch. We’ve all been there though. With any luck, our analysis has at the very least helped you and your organization understand the risk a bit better. As we dig into this more, Splunk’s Threat Research Team will work to bring more robust methods to help identify this activity. We will certainly keep our network defender hats on as more information is released and update the blog as needed. If you want to get notified of blogs like these please sign up for SURGe alerts and quarterly newsletter.