CVE-2023-5678: OpenSSL X9.42 DH Keys DoS Vulnerability

CVE-2023-5678 Overview

CVE-2023-5678 is a Denial of Service vulnerability in OpenSSL affecting the generation and validation of X9.42 Diffie-Hellman (DH) keys. Applications using the DH_generate_key() function to generate X9.42 DH keys or using DH_check_pub_key(), DH_check_pub_key_ex(), or EVP_PKEY_public_check() to validate X9.42 DH keys or parameters may experience significant delays when processing excessively large P and Q parameters from untrusted sources.

The vulnerability stems from incomplete validation checks in the DH key handling functions. While DH_check() performs all necessary checks (following fixes for CVE-2023-3817), DH_check_pub_key() lacks these protections and remains vulnerable to excessively large P and Q parameters. Similarly, DH_generate_key() checks for excessively large P values but fails to validate Q parameters.

Critical Impact
Applications accepting DH keys or parameters from untrusted sources can be forced into resource exhaustion, causing extended delays and potential service unavailability.

Affected Products

OpenSSL (all versions prior to patched releases)
OpenSSL pkey command line application (when using -pubcheck option)
OpenSSL genpkey command line application

Discovery Timeline

2023-11-06 - CVE-2023-5678 published to NVD
2025-12-02 - Last updated in NVD database

Technical Details for CVE-2023-5678

Vulnerability Analysis

This vulnerability is classified under CWE-606 (Unchecked Input for Loop Condition) and CWE-754 (Improper Check for Unusual or Exceptional Conditions). The core issue lies in how OpenSSL handles X9.42 DH key generation and validation operations when presented with unusually large cryptographic parameters.

When processing DH keys, OpenSSL performs mathematical operations whose computational complexity scales with the size of the input parameters. The affected functions fail to enforce adequate bounds checking on the P (prime modulus) and Q (subgroup order) parameters before initiating these computationally expensive operations.

An attacker can craft malicious DH parameters with excessively large values that, when processed, cause the application to enter prolonged computation cycles. This algorithmic complexity attack does not require authentication and can be triggered remotely through any application interface that accepts DH parameters.

The OpenSSL SSL/TLS implementation and the OpenSSL 3.0 and 3.1 FIPS providers are explicitly noted as not affected by this vulnerability.

Root Cause

The root cause is an improper check for unusual or exceptional conditions in OpenSSL's DH key handling functions. Specifically:

DH_check_pub_key() does not perform the necessary validation checks for excessively large P and Q parameters that were added to DH_check() following CVE-2023-3817
DH_generate_key() validates the P parameter size but omits equivalent validation for the Q parameter
These functions are also called internally by DH_check_pub_key_ex(), EVP_PKEY_public_check(), and EVP_PKEY_generate(), extending the attack surface

Attack Vector

The attack vector is network-based and requires no user interaction or privileges. An attacker can exploit this vulnerability by supplying crafted X9.42 DH keys or parameters to any application endpoint that processes DH key material from untrusted sources. The attack flow involves:

Attacker identifies an application using vulnerable OpenSSL DH functions
Attacker crafts DH parameters with excessively large P and/or Q values
Attacker submits the malicious parameters to the target application
The application's OpenSSL library attempts to process the parameters
Computational resources are exhausted during the lengthy processing operation
Service availability is degraded or denied to legitimate users

The vulnerability can also be triggered through OpenSSL command line tools when using the pkey -pubcheck option or the genpkey utility with malicious input files.

Detection Methods for CVE-2023-5678

Indicators of Compromise

Abnormally high CPU utilization during DH key processing operations
Extended response times or timeouts in applications performing key validation
Large DH parameter files or network payloads containing oversized P/Q values
Process hangs or unresponsiveness in OpenSSL-dependent services

Detection Strategies

Monitor for unusual CPU consumption spikes in processes using OpenSSL libraries
Implement application-level logging for DH key validation operations with timing metrics
Deploy network intrusion detection rules to identify oversized DH parameter payloads
Use SentinelOne's behavioral AI to detect anomalous resource consumption patterns indicative of algorithmic complexity attacks

Monitoring Recommendations

Enable detailed logging for cryptographic operations in affected applications
Set up alerts for processes exceeding normal computation time thresholds during key operations
Monitor OpenSSL library function calls using runtime application self-protection (RASP) tools
Track memory and CPU metrics for services that accept DH parameters from external sources

How to Mitigate CVE-2023-5678

Immediate Actions Required

Update OpenSSL to the latest patched version for your release branch
Audit applications to identify those using affected DH functions with untrusted input
Implement input validation to reject DH parameters exceeding reasonable size thresholds
Consider rate limiting for endpoints accepting DH key material

Patch Information

OpenSSL has released security patches addressing this vulnerability. The fixes add proper bounds checking for both P and Q parameters across all affected functions. Patches are available through:

OpenSSL Security Advisory 20231106 - Official security advisory with remediation guidance
OpenSSL Commit Diff db925ae2 - Source code patch
OpenSSL Commit Diff ddeb4b6 - Additional commit fix

Downstream vendors have also released advisories, including NetApp Security Advisory ntap-20231130-0010 and Debian LTS announcements.

Workarounds

Validate DH parameter sizes at the application layer before passing to OpenSSL functions
Implement timeouts for cryptographic operations to prevent indefinite processing
Use alternative key exchange mechanisms where X9.42 DH is not strictly required
Restrict network access to services accepting DH parameters to trusted sources only

bash

# Example: Check OpenSSL version and upgrade
openssl version
# Upgrade via package manager (example for Debian/Ubuntu)
sudo apt update && sudo apt upgrade openssl libssl-dev
# Verify updated version
openssl version