CVE-2026-63074: OpenSSL CMP Memory Exhaustion DOS Vulnerability

CVE-2026-63074 Overview

CVE-2026-63074 is a denial-of-service vulnerability in the OpenSSL Certificate Management Protocol (CMP) implementation. The CMP code caches additional certificates (extraCerts) received in incoming messages but never removes them when a message is rejected. A remote attacker who repeatedly sends CMP messages containing unique extraCerts can drive unbounded memory growth in servers that reuse a single OSSL_CMP_CTX across many requests. The condition is tracked as [CWE-770: Allocation of Resources Without Limits or Throttling]. The FIPS module is not affected because CMP lives outside the FIPS boundary.

Critical Impact
Long-running CMP servers reusing a shared context can exhaust memory, producing out-of-memory (OOM) conditions and service outages.

Affected Products

OpenSSL builds that expose the CMP client/server APIs via crypto/cmp/
Applications reusing a single OSSL_CMP_CTX for the lifetime of a server process
CMP servers accepting untrusted or unauthenticated peer messages

Discovery Timeline

2026-08-25 - CVE-2026-63074 published to the National Vulnerability Database
2026-08-25 - OpenSSL security advisory published (OpenSSL Security Advisory 20260825)
2026-08-25 - Last updated in NVD database

Technical Details for CVE-2026-63074

Vulnerability Analysis

The OpenSSL CMP verifier appends every certificate delivered in a peer's extraCerts field to the context's untrusted certificate stack. Before the fix, the code removed those additions only when the context was explicitly configured with noCacheExtraCerts. If the message was rejected during validation, the appended certificates remained resident in the OSSL_CMP_CTX untrusted store.

A malicious client can craft rejected CMP messages that each carry fresh, distinct certificates. Every request permanently grows the untrusted stack held by the server context. Because the growth is bounded only by peer request volume, sustained traffic leads to memory exhaustion and process termination.

The vulnerability is a pure resource-management flaw. It does not expose confidentiality or integrity, but a network-reachable attacker with no authentication can trigger it. Exploitation complexity is elevated because the attacker must generate large numbers of unique certificates to see measurable impact.

Root Cause

The cleanup path in crypto/cmp/cmp_vfy.c was gated exclusively on the noCacheExtraCerts flag. When message validation failed, the code left the previously appended extraCerts in ctx->untrusted. Combined with server designs that reuse one OSSL_CMP_CTX for the lifetime of the process, this produced a monotonically growing certificate stack under attacker control.

Attack Vector

An unauthenticated remote attacker sends a stream of CMP messages to the target server. Each message carries an extraCerts list populated with newly generated, unique certificates and is structured so that the server rejects the message during header or signature validation. The rejection path retains the certificates. Repeating the request drives the untrusted stack toward unbounded growth and eventual OOM.

// Patch: crypto/cmp/cmp_vfy.c
// Source: https://github.com/openssl/openssl/commit/01e567978a55fba18142a230380c31296049fae7
     res = 1; /* support more aggressive fuzzing by letting invalid msg pass */
 #endif

-    /* remove extraCerts again if not caching */
-    if (ctx->noCacheExtraCerts)
+    /*
+     * remove extraCerts again if not caching
+     * or if we failed validation above, lest a remote user
+     * starts sending us lots of certificates in invalid messages
+     * leading to a DOS from unbounded certificate stack growth
+     */
+    if (ctx->noCacheExtraCerts || res != 1)
         while (num_added-- > 0)
             X509_free(sk_X509_shift(ctx->untrusted));

The fix expands the cleanup condition to also trigger when validation fails (res != 1), so rejected messages no longer leave residue in the untrusted stack. See also OpenSSL commit 74ae7f6 and OpenSSL commit 21a5d96.

Detection Methods for CVE-2026-63074

Indicators of Compromise

Sustained inbound CMP traffic (typically TCP/829 or HTTP-transported CMP) from a single source or small set of sources
Repeated CMP request/response pairs where the server returns PKIStatus rejections
Resident set size (RSS) of the CMP server process climbing linearly with request count
OOM kills or allocator failures in logs correlated with peaks in CMP request volume

Detection Strategies

Instrument the CMP application to track the depth of ctx->untrusted and alert on abnormal growth
Correlate rejected CMP messages with per-source request rates to identify abusive clients
Baseline process memory for CMP daemons and alert on deviations that outpace legitimate enrollment cycles

Monitoring Recommendations

Ingest CMP server logs and process memory metrics into a SIEM or observability platform for correlation
Track counts of validation failures by peer identity and source IP to surface flood patterns
Monitor for repeated OOM restarts on hosts running CMP endpoints, since these often precede sustained outage

How to Mitigate CVE-2026-63074

Immediate Actions Required

Upgrade OpenSSL to a patched release incorporating the fixes referenced in the OpenSSL Security Advisory 20260825
Restart CMP server processes after patching to release accumulated extraCerts from long-lived contexts
Restrict network exposure of CMP endpoints to trusted registration authorities and known clients until patched

Patch Information

OpenSSL addressed the issue by extending the cleanup path in crypto/cmp/cmp_vfy.c to remove appended extraCerts whenever message validation fails, matching the behavior used when caching is disabled. The relevant upstream commits are 01e5679, 21a5d96, 74ae7f6, 75360af, and f636f9c.

Workarounds

Configure the CMP context with OSSL_CMP_CTX_set_option(ctx, OSSL_CMP_OPT_NO_CACHE_EXTRACERTS, 1) so extra certificates are removed after every message
Recycle the OSSL_CMP_CTX per request or per short-lived worker instead of reusing one context for the process lifetime
Rate-limit CMP requests per source and cap process memory using systemdMemoryMax, cgroups, or container limits to constrain blast radius

bash

# Example: cap CMP daemon memory with systemd to prevent host-wide OOM
# /etc/systemd/system/cmp-server.service.d/override.conf
[Service]
MemoryMax=1G
MemoryHigh=768M
Restart=on-failure
RestartSec=5s