CVE-2026-63075: OpenSSL QUIC Stack DOS Vulnerability

CVE-2026-63075 Overview

CVE-2026-63075 is a denial-of-service vulnerability in the OpenSSL QUIC stack. The flaw involves unbounded resource allocation [CWE-770] in the acknowledgment manager (ACKM) component. When OpenSSL sends an ACK-only packet, it retains packet metadata even though QUIC does not require peers to acknowledge such packets. A remote peer that completes a QUIC handshake can drive the OpenSSL implementation into transmitting ACK-only packets repeatedly (by sending PING frames) while withholding acknowledgments for ack-eliciting data. This causes connection-scoped memory to grow for the lifetime of the connection, exhausting resources on the server.

Critical Impact
A remote attacker who can complete a QUIC handshake can trigger sustained memory growth per connection, causing denial of service through memory exhaustion on OpenSSL QUIC endpoints.

Affected Products

OpenSSL versions containing the QUIC stack (non-FIPS module)
Applications and services embedding OpenSSL QUIC for TLS/QUIC transport
Servers accepting untrusted QUIC connections using OpenSSL

Discovery Timeline

2026-08-25 - OpenSSL Security Advisory 20260825 published
2026-08-25 - CVE-2026-63075 published to NVD
2026-08-25 - Last updated in NVD database

Technical Details for CVE-2026-63075

Vulnerability Analysis

The vulnerability resides in the OpenSSL QUIC acknowledgment manager implemented in ssl/quic/quic_ackm.c. When the OpenSSL QUIC stack transmits a packet containing only ACK frames, it stores full transmission metadata in the TX packet history structure. QUIC's design does not require peers to acknowledge ACK-only packets, so those history entries have no natural retirement path when the peer refuses to send acknowledgments for other traffic. An attacker with a valid QUIC handshake can force the OpenSSL peer to emit ACK-only packets by sending PING frames, then withhold acknowledgments for legitimate ack-eliciting data. This blocks the normal cleanup of the TX history and drives connection-scoped memory growth. The OpenSSL FIPS module is not affected because the QUIC code lies outside the FIPS module boundary.

Root Cause

The root cause is unbounded metadata retention in the ACKM TX history for packets that are not ack-eliciting and not in-flight. The original ossl_ackm_on_tx_packet() path treated all transmitted packets uniformly, storing metadata regardless of whether the packet could ever be acknowledged. Without a bound on history size or a fast path for ACK-only packets, sustained adversarial traffic causes linear memory growth per connection.

Attack Vector

Exploitation requires network reachability and a completed QUIC handshake. The attacker sends repeated PING frames to force OpenSSL to emit ACK-only responses, then withholds any acknowledgments for ack-eliciting data sent by the server. Repeating this pattern, potentially across many concurrent connections, drives sustained memory consumption until the process exhausts available memory.

// Patch: include/internal/quic_ackm.h
// New API to record transmission of ACK-only packets separately
int ossl_ackm_on_tx_packet(OSSL_ACKM *ackm, OSSL_ACKM_TX_PKT *pkt);

/*
 * Records transmission of a packet containing only ACK frames.
 */
int ossl_ackm_on_tx_ack_only_packet(OSSL_ACKM *ackm, OSSL_ACKM_TX_PKT *pkt);
int ossl_ackm_on_rx_datagram(OSSL_ACKM *ackm, size_t num_bytes);

// Patch: ssl/quic/quic_ackm.c
int ossl_ackm_on_tx_ack_only_packet(OSSL_ACKM *ackm, OSSL_ACKM_TX_PKT *pkt)
{
    struct tx_pkt_history_st *h;
    unsigned int pkt_space;

    if (pkt == NULL || pkt->pkt_space >= QUIC_PN_SPACE_NUM)
        return 0;

    /*
     * A packet containing only an ACK frame must not be treated as
     * in-flight or ack-eliciting; otherwise ossl_ackm_on_tx_packet()
     * would perform bytes-in-flight/timer/CC bookkeeping for a packet
     * we are about to discard from history.
     */
    if (pkt->is_inflight || pkt->is_ack_eliciting)
        return 0;

    pkt_space = pkt->pkt_space;
    h = get_tx_history(ackm, pkt_space);
    // Watermarks updated without storing full metadata
}
// Source: https://github.com/openssl/openssl/commit/7308946576b12e64b8be53bcf0a120354b2b42bc

Detection Methods for CVE-2026-63075

Indicators of Compromise

Sustained memory growth in processes linked against a vulnerable OpenSSL QUIC build, correlated with active QUIC sessions on UDP/443 or other QUIC listeners.
Long-lived QUIC connections from a single peer sending high volumes of PING frames while sending few or no ACK frames.
Out-of-memory (OOM) terminations of QUIC-enabled services under otherwise modest connection counts.

Detection Strategies

Monitor per-process resident set size (RSS) and heap growth for services that terminate QUIC using OpenSSL, alerting on unbounded growth over minutes to hours.
Inspect QUIC telemetry for asymmetric frame ratios where inbound PING frames are high but inbound ACK coverage of server-sent data is low.
Correlate connection duration and byte counts against memory consumption to identify low-traffic connections consuming disproportionate memory.

Monitoring Recommendations

Ingest network flow logs and OpenSSL debug/trace output into a centralized analytics platform to baseline normal QUIC ACK behavior.
Alert on repeated OOM-killer events or cgroup memory pressure on QUIC-fronted services.
Track OpenSSL library versions across the fleet to identify hosts still running vulnerable QUIC builds.

How to Mitigate CVE-2026-63075

Immediate Actions Required

Inventory all systems using OpenSSL with QUIC enabled and identify Internet-exposed listeners.
Apply the OpenSSL security update referenced in OpenSSL Security Advisory 20260825 as soon as vendor packages become available.
Enforce per-connection memory limits, connection rate limits, and idle timeouts on QUIC endpoints to bound attacker impact until patches are deployed.

Patch Information

The fix updates the ACKM to account for ACK-only packet transmission in the TX history watermarks without retaining full packet metadata. It is implemented across commits 7308946, 7c98d79, bf84721, and c902e5f. Rebuild or upgrade OpenSSL to a version that incorporates the new ossl_ackm_on_tx_ack_only_packet() code path.

Workarounds

Disable QUIC transport in affected applications where feasible and fall back to TCP-based TLS until patches are deployed.
Restrict access to QUIC listeners at the network edge, allowing only trusted client ranges where possible.
Configure aggressive QUIC idle timeouts and per-source connection caps to limit sustained abuse of individual connections.

bash

# Example: block untrusted UDP/443 (QUIC) at the edge until patched
iptables -A INPUT -p udp --dport 443 -m conntrack --ctstate NEW \
  -m hashlimit --hashlimit-name quic_new \
  --hashlimit-above 20/sec --hashlimit-mode srcip -j DROP

# Verify installed OpenSSL version on Linux hosts
openssl version -a