CVE-2026-54874: OpenSSL DTLS Memory Exhaustion DoS Vulnerability

CVE-2026-54874 Overview

CVE-2026-54874 is a memory amplification vulnerability in OpenSSL's Datagram Transport Layer Security (DTLS) implementation. When a DTLS handshake is in progress, an attacker can send small forged records claiming to belong to the next epoch. OpenSSL buffers these records but retains the entire read buffer (approximately 16 kilobytes) for each one, rather than only the record bytes. Up to 100 such records may be buffered per connection, causing an endpoint to retain around 1.7 megabytes of memory per association. The classification is [CWE-405: Asymmetric Resource Consumption (Amplification)].

Critical Impact
An attacker achieves an approximate 1200x memory amplification factor per DTLS connection, enabling remote memory exhaustion denial-of-service against DTLS servers.

Affected Products

OpenSSL 4.0 (fixed in 4.0.2)
OpenSSL 3.6, 3.5, 3.4, 3.0 (fixed in 3.6.4, 3.5.8, 3.4.7, 3.0.22)
OpenSSL 1.1.1 and 1.0.2 (premium support: 1.1.1zi and 1.0.2zr)

Discovery Timeline

18 May 2026 - Vulnerability reported by Amazon Web Services
25 August 2026 - OpenSSL Security Advisory 20260825 published and fix released by Matt Caswell
2026-08-25 - CVE-2026-54874 published to NVD
2026-08-25 - Last updated in NVD database

Technical Details for CVE-2026-54874

Vulnerability Analysis

DTLS runs over UDP, which permits packet reordering. During a handshake, a peer may legitimately advance to the next epoch by sending ChangeCipherSpec and Finished messages before the local endpoint has processed the transition. OpenSSL buffers those early records for later processing once the local endpoint catches up.

The defect lies in how each buffered record is stored. The code in ssl/record/methods/dtls_meth.c retains the entire TLS_BUFFER read buffer (sized for the largest possible DTLS record, roughly 16 KB) rather than copying only the record's actual bytes. With up to 100 records buffered per connection, memory usage grows to about 1.7 MB per association from only a few kilobytes of attacker traffic.

Root Cause

The root cause is an inefficient buffer retention pattern. When queuing a next-epoch record, OpenSSL transferred ownership of the full read buffer to the queued item and allocated a fresh read buffer for subsequent reads. This produced a constant per-record allocation of the maximum record size regardless of the actual payload length.

Attack Vector

A remote unauthenticated peer initiates a DTLS handshake against a vulnerable server, then transmits a stream of small forged records with an epoch value ahead of the current one. Each record causes the server to retain a full read buffer. Multiplying the effect across many concurrent associations yields remote memory exhaustion. FIPS modules are not affected because the vulnerable code is outside the FIPS module boundary.

// Patch: ssl/record/methods/dtls_meth.c
// Copies only the record's on-wire bytes instead of the entire read buffer
         return -1;
     }
 
-    rdata->packet = rl->packet;
+    /*
+     * Take a copy of just this record's on-wire bytes (header + ciphertext)
+     * rather than the whole (much larger) read buffer. The live rl->rbuf is
+     * left untouched and continues to be used for subsequent reads.
+     */
     rdata->packet_length = rl->packet_length;
-    memcpy(&(rdata->rbuf), &rl->rbuf, sizeof(TLS_BUFFER));
-    memcpy(&(rdata->rrec), &rl->rrec[0], sizeof(TLS_RL_RECORD));
-
-    item->data = rdata;
-
-    rl->packet = NULL;
-    rl->packet_length = 0;
-    memset(&rl->rbuf, 0, sizeof(TLS_BUFFER));
-    memset(&rl->rrec[0], 0, sizeof(rl->rrec[0]));
-
-    if (!tls_setup_read_buffer(rl)) {
-        /* RLAYERfatal() already called */
-        OPENSSL_free(rdata->rbuf.buf);
+    rdata->packet = OPENSSL_memdup(rl->packet, rl->packet_length);
+    if (rdata->packet == NULL) {
         OPENSSL_free(rdata);
         pitem_free(item);
+        RLAYERfatal(rl, SSL_AD_INTERNAL_ERROR, ERR_R_CRYPTO_LIB);
         return -1;
     }

Source: OpenSSL commit 4808b5d

Detection Methods for CVE-2026-54874

Indicators of Compromise

Sustained rise in resident memory of processes linking libssl that expose DTLS listeners, without a matching rise in throughput.
High volume of small inbound UDP datagrams to DTLS service ports (for example, 4433, 5061 for SIP, or WebRTC media ports) during in-progress handshakes.
DTLS records with epoch values ahead of the current negotiated epoch appearing in packet captures.

Detection Strategies

Inventory servers using OpenSSL DTLS by querying installed package versions and comparing against the fixed releases (3.0.22, 3.4.7, 3.5.8, 3.6.4, 4.0.2).
Instrument DTLS endpoints with per-connection memory accounting and alert when buffered next-epoch record count approaches the 100-record ceiling.
Baseline UDP source diversity per DTLS endpoint and flag single-source or low-source floods of partial handshakes.

Monitoring Recommendations

Track RSS and heap growth for DTLS-serving processes and correlate with active DTLS association counts.
Log DTLS handshake state transitions and count records queued for future epochs per connection.
Monitor UDP session tables for elevated half-open DTLS associations from single peers.

How to Mitigate CVE-2026-54874

Immediate Actions Required

Upgrade OpenSSL to a fixed release: 4.0.2, 3.6.4, 3.5.8, 3.4.7, or 3.0.22. Premium support customers on 1.1.1 and 1.0.2 should move to 1.1.1zi and 1.0.2zr.
Rebuild or repackage any statically linked applications and DTLS-enabled services (VPN concentrators, SIP/WebRTC gateways, IoT brokers) against the patched libraries.
Restart services after the upgrade to ensure the new library is loaded.

Patch Information

The fix, developed by Matt Caswell, changes DTLS next-epoch record queuing so only the record's on-wire bytes are copied via OPENSSL_memdup, leaving the live read buffer intact. Full details are in the OpenSSL Security Advisory 20260825 and the associated commits: 4808b5d, 7110cb2f, cc0c671, and f52ffc1.

Workarounds

Enforce strict caps on concurrent DTLS associations per source IP at the network edge to bound total exposure.
Apply UDP rate limiting to DTLS listener ports to reduce forged next-epoch record floods.
Where feasible, front DTLS services with a session-aware proxy that validates handshake progression before forwarding records.

bash

# Verify installed OpenSSL version and confirm patched release
openssl version -a

# Debian/Ubuntu
sudo apt-get update && sudo apt-get install --only-upgrade openssl libssl3

# RHEL/CentOS/Fedora
sudo dnf upgrade openssl openssl-libs

# Restart DTLS-dependent services (example)
sudo systemctl restart openvpn strongswan coturn