CVE-2023-6237: OpenSSL RSA Key Validation DoS Vulnerability

CVE-2023-6237 Overview

CVE-2023-6237 is a Denial of Service vulnerability in OpenSSL affecting the EVP_PKEY_public_check() function when processing RSA public keys. Applications that use this function to validate RSA public keys from untrusted sources may experience significant delays when processing excessively long invalid RSA public keys, potentially leading to service disruption.

When EVP_PKEY_public_check() is called on RSA public keys, a computation is performed to confirm that the RSA modulus (n) is composite. For valid RSA keys, n is a product of two or more large primes, and this computation completes quickly. However, if n is an overly large prime, the computation takes an excessive amount of time, creating the denial of service condition.

Critical Impact

Applications processing RSA public keys from untrusted sources using EVP_PKEY_public_check() are vulnerable to algorithmic complexity attacks that can cause extended delays and service unavailability.

Affected Products

• OpenSSL 3.0 (including FIPS provider)
• OpenSSL 3.1 (including FIPS provider)
• OpenSSL 3.2

Discovery Timeline

• 2024-01-15 - OpenSSL Security Advisory published
• 2024-04-25 - CVE CVE-2023-6237 published to NVD
• 2024-11-21 - Last updated in NVD database

Technical Details for CVE-2023-6237

Vulnerability Analysis

The vulnerability exists in the RSA public key validation logic within OpenSSL. When the EVP_PKEY_public_check() function is invoked, it performs a primality check on the RSA modulus to ensure it is composite (a product of prime numbers). This is a standard validation step for RSA keys since the security of RSA depends on the modulus being a product of large primes.

The problem arises when an attacker supplies a crafted RSA key where the modulus n is itself an extremely large prime number rather than a composite. The primality testing algorithm used by OpenSSL has computational complexity that scales poorly with very large prime inputs, causing the validation to take an inordinate amount of time.

Importantly, the OpenSSL SSL/TLS implementation is not affected by this vulnerability. The EVP_PKEY_public_check() function is not called from other internal OpenSSL functions. However, the OpenSSL pkey command-line application is vulnerable when used with the -pubin and -check options on untrusted data.

Root Cause

The root cause is CWE-606: Unchecked Input for Loop Condition. The RSA key validation code did not impose an upper bound on the modulus size before performing the computationally expensive primality check. Without this limit, an attacker could supply keys with arbitrarily large modulus values, causing the algorithm to run for an extended period.

Attack Vector

An attacker can exploit this vulnerability by supplying a maliciously crafted RSA public key with an excessively large prime number as the modulus to any application that:

1. Accepts RSA public keys from untrusted sources
2. Validates those keys using the EVP_PKEY_public_check() function

The attack is network-accessible but requires specific conditions (high attack complexity) - the target application must explicitly call the vulnerable function on attacker-controlled input.

// Security patch in crypto/rsa/rsa_sp800_56b_check.c - Limit the execution time of RSA public key check
         return 0;
 
     nbits = BN_num_bits(rsa->n);
+    if (nbits > OPENSSL_RSA_MAX_MODULUS_BITS) {
+        ERR_raise(ERR_LIB_RSA, RSA_R_MODULUS_TOO_LARGE);
+        return 0;
+    }
+
 #ifdef FIPS_MODULE
     /*
      * (Step a): modulus must be 2048 or 3072 (caveat from SP800-56Br1)

Source: GitHub OpenSSL Commit

The patch adds a check against OPENSSL_RSA_MAX_MODULUS_BITS before performing the expensive primality computation, effectively limiting the maximum size of RSA modulus that will be processed.

Detection Methods for CVE-2023-6237

Indicators of Compromise

• Unusual CPU spikes during RSA key validation operations
• Extended processing times for key verification requests
• Application timeouts or hangs when processing public key submissions
• Increased memory consumption during cryptographic operations

Detection Strategies

• Monitor application performance metrics for anomalous delays in key validation functions
• Implement logging around EVP_PKEY_public_check() calls to track processing duration
• Review application code for instances where RSA public keys from untrusted sources are validated
• Deploy application-level timeouts for cryptographic operations

Monitoring Recommendations

• Set up alerts for abnormal CPU utilization in services handling cryptographic operations
• Monitor OpenSSL library usage and version across infrastructure
• Track response times for endpoints that accept and process public keys
• Implement circuit breakers for cryptographic validation operations

How to Mitigate CVE-2023-6237

Immediate Actions Required

• Upgrade OpenSSL to patched versions: 3.0.13, 3.1.5, or 3.2.1
• Audit applications for usage of EVP_PKEY_public_check() with untrusted input
• Implement input validation to reject oversized RSA keys before validation
• Apply rate limiting to endpoints that process public key submissions

Patch Information

OpenSSL has released security patches addressing this vulnerability. The fix adds a check for the modulus bit length against OPENSSL_RSA_MAX_MODULUS_BITS before performing the primality test. Multiple commits have been published for different OpenSSL branches:

• OpenSSL 3.2 fix
• OpenSSL 3.1 fix
• OpenSSL 3.0 fix

Additional vendor advisories are available from NetApp.

Workarounds

• Avoid calling EVP_PKEY_public_check() on RSA keys from untrusted sources if the check is not strictly necessary
• Implement application-level validation to check RSA modulus size before calling OpenSSL functions
• Add timeouts around key validation operations to prevent indefinite blocking
• Use the OpenSSL pkey command-line tool only with trusted key sources

# Check OpenSSL version and upgrade if vulnerable
openssl version
# Upgrade to patched version (example for Debian/Ubuntu)
sudo apt update && sudo apt upgrade openssl
# Verify new version
openssl version