CVE-2023-5678: OpenSSL X9.42 DH Keys DoS Vulnerability

CVE-2023-5678 Overview

CVE-2023-5678 is a Denial of Service vulnerability in OpenSSL affecting the generation and validation of X9.42 Diffie-Hellman (DH) keys. Applications using the DH_generate_key() function to generate X9.42 DH keys or using DH_check_pub_key(), DH_check_pub_key_ex(), or EVP_PKEY_public_check() to validate X9.42 DH keys or parameters may experience significant delays when processing excessively large P and Q parameters from untrusted sources.

The vulnerability stems from incomplete validation checks in the DH key handling functions. While DH_check() performs all necessary checks (following fixes for CVE-2023-3817), DH_check_pub_key() lacks these protections and remains vulnerable to excessively large P and Q parameters. Similarly, DH_generate_key() checks for excessively large P values but fails to validate Q parameters.

Critical Impact

Applications accepting DH keys or parameters from untrusted sources can be forced into resource exhaustion, causing extended delays and potential service unavailability.

Affected Products

• OpenSSL (all versions prior to patched releases)
• OpenSSL pkey command line application (when using -pubcheck option)
• OpenSSL genpkey command line application

Discovery Timeline

• 2023-11-06 - CVE-2023-5678 published to NVD
• 2025-12-02 - Last updated in NVD database

Technical Details for CVE-2023-5678

Vulnerability Analysis

This vulnerability is classified under CWE-606 (Unchecked Input for Loop Condition) and CWE-754 (Improper Check for Unusual or Exceptional Conditions). The core issue lies in how OpenSSL handles X9.42 DH key generation and validation operations when presented with unusually large cryptographic parameters.

When processing DH keys, OpenSSL performs mathematical operations whose computational complexity scales with the size of the input parameters. The affected functions fail to enforce adequate bounds checking on the P (prime modulus) and Q (subgroup order) parameters before initiating these computationally expensive operations.

An attacker can craft malicious DH parameters with excessively large values that, when processed, cause the application to enter prolonged computation cycles. This algorithmic complexity attack does not require authentication and can be triggered remotely through any application interface that accepts DH parameters.

The OpenSSL SSL/TLS implementation and the OpenSSL 3.0 and 3.1 FIPS providers are explicitly noted as not affected by this vulnerability.

Root Cause

The root cause is an improper check for unusual or exceptional conditions in OpenSSL's DH key handling functions. Specifically:

1. DH_check_pub_key() does not perform the necessary validation checks for excessively large P and Q parameters that were added to DH_check() following CVE-2023-3817
2. DH_generate_key() validates the P parameter size but omits equivalent validation for the Q parameter
3. These functions are also called internally by DH_check_pub_key_ex(), EVP_PKEY_public_check(), and EVP_PKEY_generate(), extending the attack surface

Attack Vector

The attack vector is network-based and requires no user interaction or privileges. An attacker can exploit this vulnerability by supplying crafted X9.42 DH keys or parameters to any application endpoint that processes DH key material from untrusted sources. The attack flow involves:

1. Attacker identifies an application using vulnerable OpenSSL DH functions
2. Attacker crafts DH parameters with excessively large P and/or Q values
3. Attacker submits the malicious parameters to the target application
4. The application's OpenSSL library attempts to process the parameters
5. Computational resources are exhausted during the lengthy processing operation
6. Service availability is degraded or denied to legitimate users

The vulnerability can also be triggered through OpenSSL command line tools when using the pkey -pubcheck option or the genpkey utility with malicious input files.

Detection Methods for CVE-2023-5678

Indicators of Compromise

• Abnormally high CPU utilization during DH key processing operations
• Extended response times or timeouts in applications performing key validation
• Large DH parameter files or network payloads containing oversized P/Q values
• Process hangs or unresponsiveness in OpenSSL-dependent services

Detection Strategies

• Monitor for unusual CPU consumption spikes in processes using OpenSSL libraries
• Implement application-level logging for DH key validation operations with timing metrics
• Deploy network intrusion detection rules to identify oversized DH parameter payloads
• Use SentinelOne's behavioral AI to detect anomalous resource consumption patterns indicative of algorithmic complexity attacks

Monitoring Recommendations

• Enable detailed logging for cryptographic operations in affected applications
• Set up alerts for processes exceeding normal computation time thresholds during key operations
• Monitor OpenSSL library function calls using runtime application self-protection (RASP) tools
• Track memory and CPU metrics for services that accept DH parameters from external sources

How to Mitigate CVE-2023-5678

Immediate Actions Required

• Update OpenSSL to the latest patched version for your release branch
• Audit applications to identify those using affected DH functions with untrusted input
• Implement input validation to reject DH parameters exceeding reasonable size thresholds
• Consider rate limiting for endpoints accepting DH key material

Patch Information

OpenSSL has released security patches addressing this vulnerability. The fixes add proper bounds checking for both P and Q parameters across all affected functions. Patches are available through:

• OpenSSL Security Advisory 20231106 - Official security advisory with remediation guidance
• OpenSSL Commit Diff db925ae2 - Source code patch
• OpenSSL Commit Diff ddeb4b6 - Additional commit fix

Downstream vendors have also released advisories, including NetApp Security Advisory ntap-20231130-0010 and Debian LTS announcements.

Workarounds

• Validate DH parameter sizes at the application layer before passing to OpenSSL functions
• Implement timeouts for cryptographic operations to prevent indefinite processing
• Use alternative key exchange mechanisms where X9.42 DH is not strictly required
• Restrict network access to services accepting DH parameters to trusted sources only

# Example: Check OpenSSL version and upgrade
openssl version
# Upgrade via package manager (example for Debian/Ubuntu)
sudo apt update && sudo apt upgrade openssl libssl-dev
# Verify updated version
openssl version