CVE-2023-3817: OpenSSL DH Key Check DoS Vulnerability

CVE-2023-3817 Overview

CVE-2023-3817 is a Denial of Service vulnerability affecting OpenSSL's Diffie-Hellman (DH) parameter validation functions. Applications that use the functions DH_check(), DH_check_ex(), or EVP_PKEY_param_check() to verify DH keys or parameters may experience significant delays when processing excessively long DH keys or parameters. When the key or parameters being checked have been obtained from an untrusted source, this may lead to a Denial of Service condition.

This vulnerability was discovered as a follow-up to CVE-2023-3446, where it was identified that a large q parameter value can trigger an overly long computation during certain validation checks. A correct q value cannot be larger than the modulus p parameter, making these checks unnecessary when q exceeds p.

Critical Impact

Applications calling DH parameter validation functions with untrusted input can be forced into an unresponsive state through algorithmic complexity attacks, causing service disruption.

Affected Products

• OpenSSL 1.0.2 through 1.0.2zh (all releases in the 1.0.2 branch)
• OpenSSL 1.1.1 through 1.1.1u (all releases in the 1.1.1 branch)
• OpenSSL 3.0.x and 3.1.x versions (SSL/TLS implementation and FIPS providers are NOT affected)

Discovery Timeline

• July 31, 2023 - CVE-2023-3817 published to NVD
• May 5, 2025 - Last updated in NVD database

Technical Details for CVE-2023-3817

Vulnerability Analysis

The vulnerability exists in OpenSSL's DH_check() function and related APIs that perform validation on Diffie-Hellman parameters. When processing DH parameters with excessively large q values, the validation logic enters computationally expensive operations that can consume significant CPU resources.

The DH_check() function is designed to perform various security checks on DH parameters to ensure they are cryptographically sound. However, the function failed to perform an early validation that the q parameter does not exceed the p parameter before initiating resource-intensive checks. Since a mathematically correct q value can never be larger than p, any such input is inherently invalid and should be rejected immediately.

This vulnerability affects multiple OpenSSL functions and command-line tools:

• DH_check() - Primary affected function
• DH_check_ex() - Wrapper function that calls DH_check()
• EVP_PKEY_param_check() - Higher-level API that invokes DH checks
• dhparam command with -check option
• pkeyparam command with -check option

Importantly, the OpenSSL SSL/TLS implementation is NOT affected by this issue, as it does not process untrusted DH parameters through these functions during normal handshake operations. Additionally, the OpenSSL 3.0 and 3.1 FIPS providers are not affected.

Root Cause

The root cause is an algorithmic complexity vulnerability (CWE-834: Excessive Iteration, CWE-606: Unchecked Input for Loop Condition) in the DH parameter validation logic. The DH_check() function did not include a preliminary bounds check to verify that the q parameter is not larger than p before performing computationally expensive validation operations. By supplying a maliciously crafted DH parameter set with an oversized q value, an attacker can force the validation routines into prolonged computation cycles.

Attack Vector

An attacker can exploit this vulnerability by providing crafted DH parameters with an excessively large q value to any application that validates untrusted DH keys or parameters. The attack is network-accessible and requires no authentication or user interaction.

Attack scenarios include:

• Applications that accept and validate DH parameters from external sources
• Services that process cryptographic key material from untrusted clients
• Certificate or key validation utilities processing malicious input
• Command-line tools (dhparam, pkeyparam) when used with the -check option on untrusted input

When processing such malformed parameters, the vulnerable functions enter computationally expensive validation loops, causing the application to become unresponsive or consume excessive CPU resources.

Detection Methods for CVE-2023-3817

Indicators of Compromise

• Abnormal CPU consumption by applications using OpenSSL cryptographic functions
• Application hangs or timeouts during DH parameter validation operations
• Increased processing time for cryptographic operations involving key validation
• System resource exhaustion on servers processing external DH parameters

Detection Strategies

• Monitor for processes with unusually high CPU utilization that are linked to OpenSSL libraries
• Implement application-level logging to track DH parameter validation operation durations
• Deploy network monitoring to detect malformed or oversized DH parameters in incoming traffic
• Use SentinelOne's behavioral analysis to identify applications exhibiting DoS-like resource consumption patterns

Monitoring Recommendations

• Configure alerts for sustained high CPU usage by applications known to use OpenSSL DH validation
• Establish baseline metrics for cryptographic operation timing and alert on significant deviations
• Monitor system logs for application crashes or timeouts in services handling external cryptographic material
• Implement rate limiting on interfaces that accept DH parameters from untrusted sources

How to Mitigate CVE-2023-3817

Immediate Actions Required

• Upgrade OpenSSL to patched versions: 3.1.2, 3.0.10, 1.1.1v, or 1.0.2zi (or later)
• Identify all applications and services using affected OpenSSL versions in your environment
• Prioritize patching for services that process DH parameters from untrusted sources
• Review application code for calls to DH_check(), DH_check_ex(), or EVP_PKEY_param_check() with external input

Patch Information

OpenSSL has released security patches addressing this vulnerability. The fix adds a preliminary check to ensure that the q parameter does not exceed the p parameter before performing expensive validation operations. Patch commits are available for review:

• OpenSSL Commit for 3.1.x branch
• OpenSSL Commit for 3.0.x branch
• OpenSSL Commit for 1.1.1 branch

For complete details, refer to the OpenSSL Security Advisory.

Workarounds

• Implement input validation at the application level to reject DH parameters with q values larger than p before passing them to OpenSSL functions
• Limit or restrict the sources from which DH parameters are accepted
• Set appropriate timeouts for cryptographic operations to prevent indefinite hangs
• Consider disabling DH parameter checking for untrusted inputs until patches can be applied, understanding the security trade-offs involved

# Verify OpenSSL version after patching
openssl version -a

# Check for vulnerable functions in linked binaries
nm -D /usr/lib/x86_64-linux-gnu/libcrypto.so | grep -E "DH_check|EVP_PKEY_param_check"