CVE-2024-12797: OpenSSL RPK Authentication Bypass Flaw

CVE-2024-12797 Overview

CVE-2024-12797 is an authentication bypass vulnerability affecting OpenSSL's implementation of RFC7250 Raw Public Keys (RPKs). When clients use RPKs to authenticate a server and rely on SSL_VERIFY_PEER verification mode, the TLS/DTLS handshake may fail to abort when server authentication fails, leaving connections vulnerable to man-in-the-middle attacks.

Critical Impact

TLS and DTLS connections using raw public keys may be vulnerable to man-in-the-middle attacks when server authentication failure is not properly detected by clients.

Affected Products

• OpenSSL 3.2.x (where RPK support was initially introduced)
• OpenSSL 3.3.x
• OpenSSL 3.4.x

Discovery Timeline

• February 11, 2025 - CVE-2024-12797 published to NVD
• February 11, 2025 - OpenSSL releases security advisory
• February 18, 2025 - Last updated in NVD database

Technical Details for CVE-2024-12797

Vulnerability Analysis

This vulnerability (CWE-392: Missing Report of Error Condition) occurs in OpenSSL's Raw Public Key authentication mechanism. When a TLS client explicitly enables RPK use by the server and relies on the SSL_VERIFY_PEER verification mode to ensure server authentication, the handshake does not abort as expected when the server's RPK fails to match one of the expected public keys.

The issue is limited in scope because RPKs are disabled by default in both TLS clients and TLS servers. The vulnerability only manifests when both the client explicitly enables RPK use by the server and the server is configured to send an RPK instead of an X.509 certificate chain.

Clients that call SSL_get_verify_result() after the handshake and take appropriate action based on the returned value are not affected, as this function correctly reports the verification failure.

Root Cause

The root cause is a missing error condition report in OpenSSL's RPK implementation. When server authentication fails during the TLS/DTLS handshake with RPK verification enabled, the error condition is not properly propagated to abort the handshake. This was introduced in the initial implementation of RPK support in OpenSSL 3.2.

The FIPS modules in OpenSSL versions 3.4, 3.3, 3.2, 3.1, and 3.0 are not affected by this vulnerability.

Attack Vector

The vulnerability enables network-based man-in-the-middle attacks against TLS/DTLS connections using Raw Public Keys. An attacker positioned between a vulnerable client and server can intercept and potentially modify traffic when the client fails to detect that server authentication has failed.

The attack requires:

1. A TLS client configured to use RPK for server authentication
2. A server configured to send RPKs instead of X.509 certificates
3. The client relying solely on handshake failure for authentication verification
4. Network position enabling traffic interception

Since RPKs are not commonly deployed (disabled by default), the practical attack surface is limited to environments that have explicitly enabled this feature.

Detection Methods for CVE-2024-12797

Indicators of Compromise

• Unusual TLS connection patterns where authentication should have failed but connections were established
• Network traffic anomalies consistent with man-in-the-middle interception on RPK-enabled connections
• Applications using OpenSSL 3.2+ with RPK enabled that do not explicitly check SSL_get_verify_result()

Detection Strategies

• Audit application code for TLS implementations using RPK authentication without explicit SSL_get_verify_result() verification
• Inventory systems running OpenSSL versions 3.2.x, 3.3.x, or 3.4.x and identify those with RPK enabled
• Monitor for unexpected certificate or key negotiation behaviors in TLS handshakes

Monitoring Recommendations

• Implement network monitoring to detect potential man-in-the-middle attacks on TLS connections
• Enable verbose TLS handshake logging to identify authentication verification failures
• Review application logs for connections that complete successfully despite RPK verification issues

How to Mitigate CVE-2024-12797

Immediate Actions Required

• Update OpenSSL to the latest patched version addressing this vulnerability
• Modify client applications to explicitly call SSL_get_verify_result() after handshakes and handle verification failures appropriately
• If RPK is not required, ensure it remains disabled (default configuration)
• Audit applications using OpenSSL 3.2+ for RPK usage and verification practices

Patch Information

OpenSSL has released patches addressing this vulnerability. The following commits contain the fixes:

• OpenSSL Commit 738d4f9
• OpenSSL Commit 798779d
• OpenSSL Commit 87ebd20

For complete details, refer to the OpenSSL Security Advisory and NetApp Security Advisory.

Workarounds

• Implement explicit verification result checking by calling SSL_get_verify_result() after TLS handshakes and taking appropriate action on failures
• Disable RPK usage if not required for your application (RPK is disabled by default)
• Consider using X.509 certificate chains instead of RPKs until patches are applied
• Implement additional network security controls to detect and prevent man-in-the-middle attacks

# Verify OpenSSL version to determine if vulnerable
openssl version -a

# Check if RPK support is enabled in your configuration
# RPK is disabled by default - verify application-level configurations