CVE-2023-4807: OpenSSL POLY1305 MAC DOS Vulnerability

CVE-2023-4807 Overview

CVE-2023-4807 is a memory corruption vulnerability in OpenSSL's POLY1305 MAC (message authentication code) implementation that affects Windows 64-bit applications running on newer X86_64 processors with AVX512-IFMA instruction support. The vulnerability stems from the POLY1305 implementation failing to properly save and restore the contents of non-volatile XMM registers when calculating MACs for data larger than 64 bytes.

When exploited, the vulnerable code zeroes all XMM registers before returning to the caller instead of restoring their previous values. This internal application state corruption can lead to various consequences ranging from incorrect calculation results to application crashes resulting in denial of service. In worst-case scenarios, an attacker could potentially gain complete control of the application process, though the practical impact is limited since the registers are zeroized rather than filled with attacker-controlled data.

Critical Impact

Applications using OpenSSL on Windows 64-bit systems with AVX512-IFMA capable processors may experience state corruption, incorrect calculations, or denial of service when processing POLY1305 MAC operations on data exceeding 64 bytes.

Affected Products

• OpenSSL 1.1.1 through 1.1.1v
• OpenSSL 3.0.0 through 3.0.10
• OpenSSL 3.1.0 through 3.1.2

Discovery Timeline

• September 8, 2023 - CVE-2023-4807 published to NVD
• April 23, 2025 - Last updated in NVD database

Technical Details for CVE-2023-4807

Vulnerability Analysis

The POLY1305 MAC implementation in OpenSSL contains a critical flaw in its handling of CPU register state on the Windows 64-bit platform. The implementation uses AVX512-IFMA instructions for optimized cryptographic operations but fails to adhere to the Windows x64 calling convention, which requires certain XMM registers (XMM6-XMM15) to be preserved across function calls.

When processing data larger than 64 bytes, the vulnerable code path activates the AVX512-IFMA optimized routines. These routines utilize the non-volatile XMM registers for computational efficiency but do not save their contents before use. Upon completion, all XMM registers are indiscriminately cleared to zero rather than being restored to their original values.

The POLY1305 algorithm is most commonly used as the authentication component of the CHACHA20-POLY1305 AEAD cipher, which is widely deployed in TLS 1.2 and TLS 1.3 connections. A malicious client can influence cipher suite negotiation to force a server to use CHACHA20-POLY1305, potentially triggering the vulnerability in server applications.

Root Cause

The root cause is an improper implementation of the Windows x64 ABI (Application Binary Interface) calling convention in the AVX512-IFMA optimized POLY1305 code path. The Windows x64 calling convention designates XMM6 through XMM15 as non-volatile registers that must be preserved by called functions. The OpenSSL POLY1305 implementation violates this requirement by using these registers without saving their contents and then zeroing them instead of restoring the original values.

This constitutes a CWE-440 (Expected Behavior Violation) issue where the cryptographic library behaves contrary to the documented platform ABI requirements, leading to unpredictable behavior in calling applications that depend on the integrity of these registers.

Attack Vector

The attack vector is local, requiring the attacker to influence the application's use of the POLY1305 algorithm. The most practical attack scenario involves a malicious TLS client connecting to a vulnerable server and negotiating the use of CHACHA20-POLY1305 cipher suite during the TLS handshake.

The attack unfolds as follows: The attacker initiates a TLS connection to a server running vulnerable OpenSSL on a Windows 64-bit system with an AVX512-IFMA capable processor. The attacker's client advertises strong preference for CHACHA20-POLY1305 in the cipher suite negotiation. When the server accepts this cipher and processes authentication data larger than 64 bytes using POLY1305, the XMM register corruption occurs. Depending on how the server application uses these registers after the OpenSSL call returns, the corruption may cause calculation errors, crashes, or other undefined behavior.

Detection Methods for CVE-2023-4807

Indicators of Compromise

• Unexplained application crashes in services using OpenSSL on Windows 64-bit systems with AVX512-IFMA capable CPUs
• Calculation errors or data corruption in applications after TLS handshakes using CHACHA20-POLY1305
• System event logs showing access violations or memory corruption in OpenSSL-dependent applications

Detection Strategies

• Monitor for OpenSSL library versions in the vulnerable ranges (1.1.1 through 1.1.1v, 3.0.0 through 3.0.10, 3.1.0 through 3.1.2) on Windows 64-bit systems
• Implement software composition analysis (SCA) to identify vulnerable OpenSSL dependencies across the environment
• Check CPU capabilities for AVX512-IFMA support using system utilities to identify at-risk systems
• Review TLS configurations for CHACHA20-POLY1305 cipher suite usage on Windows servers

Monitoring Recommendations

• Deploy endpoint detection to monitor for abnormal application terminations in OpenSSL-dependent services
• Implement centralized logging to correlate potential exploitation attempts across multiple systems
• Monitor TLS handshake patterns for unusual cipher suite negotiation activity targeting CHACHA20-POLY1305

How to Mitigate CVE-2023-4807

Immediate Actions Required

• Upgrade OpenSSL to patched versions: 1.1.1w or later for 1.1.1 branch, 3.0.11 or later for 3.0 branch, 3.1.3 or later for 3.1 branch
• Apply the runtime workaround by setting the OPENSSL_ia32cap environment variable to disable AVX512-IFMA instructions
• Audit all Windows 64-bit systems with AVX512-IFMA capable processors running OpenSSL-dependent applications
• Consider temporarily disabling CHACHA20-POLY1305 cipher suites until patching is complete

Patch Information

OpenSSL has released patches addressing this vulnerability across all affected branches. The fixes ensure proper saving and restoration of non-volatile XMM registers in accordance with the Windows x64 ABI:

• OpenSSL Commit #4bfac4471 - Patch for OpenSSL 3.1.x branch
• OpenSSL Commit #6754de4a1 - Patch for OpenSSL 3.0.x branch
• OpenSSL Commit #a632d534c - Patch for OpenSSL 1.1.1 branch
• OpenSSL Security Advisory September 2023 - Official security advisory

The FIPS provider is not affected by this vulnerability.

Workarounds

• Set the OPENSSL_ia32cap environment variable to disable AVX512-IFMA instructions at runtime
• Configure TLS servers to deprioritize or disable CHACHA20-POLY1305 cipher suites as a temporary measure
• Isolate affected systems where immediate patching is not feasible

# Configuration example
# Disable AVX512-IFMA instructions to mitigate CVE-2023-4807
# Set this environment variable before starting OpenSSL-dependent applications

# On Windows (Command Prompt)
set OPENSSL_ia32cap=:~0x200000

# On Windows (PowerShell)
$env:OPENSSL_ia32cap = ":~0x200000"

# For persistent configuration, add to system environment variables
# or service startup scripts