CVE-2026-54251: Netty OHTTP Gateway DOS Vulnerability

CVE-2026-54251 Overview

CVE-2026-54251 is a memory leak vulnerability in netty-incubator-codec-ohttp, a library implementing Oblivious HTTP (OHTTP) gateway and client functionality on top of Netty. The flaw resides in the OHTTP gateway decryption path within OHttpRequestResponseContext.java. A pooled direct ByteBuf is allocated for decrypted plaintext before the Authenticated Encryption with Associated Data (AEAD) tag is verified. When decryption fails, the buffer is never released, allowing attackers to exhaust off-heap native memory through repeated invalid encrypted requests. The issue is fixed in version 0.0.23.Final.

Critical Impact
Remote unauthenticated attackers can trigger native off-heap memory exhaustion by submitting malformed OHTTP requests, rendering the gateway unable to serve legitimate traffic.

Affected Products

netty-incubator-codec-ohttp versions prior to 0.0.23.Final
OHTTP gateway deployments built on the Netty incubator OHTTP codec
Downstream services embedding the vulnerable OHTTP request/response context

Discovery Timeline

2026-09-15 - CVE-2026-54251 published to NVD
2026-09-15 - Last updated in NVD database

Technical Details for CVE-2026-54251

Vulnerability Analysis

The vulnerability is classified under [CWE-664] (Improper Control of a Resource Through its Lifetime). The decodeChunk() method in OHttpRequestResponseContext.java allocates a pooled direct ByteBuf intended to hold decrypted plaintext. This allocation occurs before the AEAD authentication tag on the ciphertext has been validated by decryptChunk().

When an attacker submits an OHTTP request with an invalid AEAD tag, decryptChunk() throws a CryptoException. Because the allocation is not wrapped in a try/finally block, the pooled ByteBuf reference is lost when the exception propagates. Netty's reference counting never reaches zero, and the underlying native memory is not returned to the pool. Each malformed request leaks a chunk of off-heap memory.

Root Cause

The root cause is missing resource cleanup on the exception path. Netty's pooled direct buffers rely on explicit release() calls to return native memory to the allocator. The original code transferred ownership only on the success path, leaving no cleanup route when decryptChunk() failed. This violates Netty's documented reference-counting contract for pooled buffers.

Attack Vector

Exploitation requires only network access to the OHTTP gateway. No authentication or user interaction is needed. An attacker crafts OHTTP-encapsulated requests with intentionally invalid AEAD tags and submits them at volume. Each request forces an allocation that is never released, gradually consuming the JVM's direct memory pool until the gateway fails with OutOfMemoryError or stops accepting requests.

java

 public void decodeChunk(ByteBufAllocator alloc, ByteBuf chunk, int chunkLength,
                         boolean completeBodyReceived, List<Object> out) throws CryptoException {
     ByteBuf decryptedChunk = alloc.buffer();
-    decryptChunk(alloc, chunk, chunkLength, completeBodyReceived, decryptedChunk);
-    binaryHttpCumulation = MERGE_CUMULATOR.cumulate(alloc, binaryHttpCumulation, decryptedChunk);
+    try {
+        decryptChunk(alloc, chunk, chunkLength, completeBodyReceived, decryptedChunk);
+        binaryHttpCumulation = MERGE_CUMULATOR.cumulate(alloc, binaryHttpCumulation, decryptedChunk);
+        // Ownership transferred to binaryHttpCumulation; do not release in finally
+        decryptedChunk = null;
+    } finally {
+        if (decryptedChunk != null) {
+            decryptedChunk.release();
+        }
+    }

Source: GitHub Commit ba6871b

Detection Methods for CVE-2026-54251

Indicators of Compromise

Progressive growth of JVM direct memory usage without corresponding increase in legitimate traffic volume.
Elevated rates of CryptoException entries in gateway logs referencing OHTTP decryption failures.
io.netty.util.internal.OutOfDirectMemoryError errors preceding gateway unavailability.
Traffic patterns showing repeated malformed OHTTP POST requests from a single source or distributed sources.

Detection Strategies

Instrument the gateway with Netty's ResourceLeakDetector at PARANOID level in test environments to confirm leak paths.
Correlate spikes in AEAD decryption failures with rising off-heap memory metrics from JVM Native Memory Tracking.
Monitor the PooledByteBufAllocator metrics for chunk allocation rates that outpace release rates.

Monitoring Recommendations

Export JVM direct memory and Netty allocator metrics to your observability platform and alert on sustained growth.
Enable structured logging for CryptoException events and forward them to your SIEM for rate-based alerting.
Track HTTP 4xx and 5xx response codes originating from OHTTP endpoints and baseline normal error volumes.

How to Mitigate CVE-2026-54251

Immediate Actions Required

Upgrade netty-incubator-codec-ohttp to version 0.0.23.Final or later across all gateway deployments.
Audit dependency trees for transitive inclusions of the vulnerable artifact and rebuild affected services.
Restart affected gateway processes to reclaim leaked native memory after patching.

Patch Information

The fix is available in release netty-incubator-codec-parent-ohttp-0.0.23.Final. The patch wraps the buffer allocation and decryption call in a try/finally block, ensuring the pooled ByteBuf is released when decryptChunk() throws. Full details are documented in GHSA-vmr9-j6wf-pmh2.

Workarounds

Deploy rate limiting in front of the OHTTP gateway to cap request volume from individual clients.
Configure JVM -XX:MaxDirectMemorySize conservatively and pair with automated restart on OOM to limit blast radius.
Place a Web Application Firewall (WAF) rule to drop OHTTP requests exceeding a reasonable failure threshold per source.

bash

# Update Maven dependency to patched version
mvn versions:use-dep-version \
  -Dincludes=io.netty.incubator:netty-incubator-codec-ohttp \
  -DdepVersion=0.0.23.Final \
  -DforceVersion=true

# Verify resolved version
mvn dependency:tree | grep netty-incubator-codec-ohttp