CVE-2026-59646: Bouncy Castle Buffer Overflow Vulnerability

CVE-2026-59646 Overview

CVE-2026-59646 is a resource allocation vulnerability in Bouncy Castle for Java affecting Datagram Transport Layer Security (DTLS) handshake processing. The DTLS handshake reassembler allocates a buffer based on an unchecked 24-bit length field supplied by the remote peer. An attacker can trigger allocations of up to 16 MiB per handshake fragment without prior authentication. The flaw is tracked under CWE-789: Memory Allocation with Excessive Size Value and can be reached over the network without user interaction.

Critical Impact
Remote, unauthenticated attackers can exhaust memory on any Java service using Bouncy Castle DTLS, degrading or halting TLS-dependent workloads.

Affected Products

Bouncy Castle for Java before 1.85
Bouncy Castle for Java LTS before 2.73.12
Bouncy Castle for Java FIPS (BC-FJA) bctls-fips before 1.0.24 (1.0.X), 2.0.24 (2.0.X), and 2.1.24 (2.1.X)

Discovery Timeline

2026-08-03 - CVE-2026-59646 published to the National Vulnerability Database (NVD)
2026-08-04 - Last updated in NVD database

Technical Details for CVE-2026-59646

Vulnerability Analysis

The vulnerability resides in the DTLS handshake reassembly path implemented in DTLSReliableHandshake. DTLS handshake messages carry a 24-bit length field that describes the full size of a fragmented handshake message. Bouncy Castle used this attacker-controlled length to size the reassembly buffer before verifying whether the value was reasonable for the negotiated context.

Because a 24-bit field can encode values up to 0xFFFFFF (16,777,215 bytes), a single crafted ClientHello or handshake fragment can force the server or client to allocate roughly 16 MiB of heap. Repeated fragments across concurrent connections amplify this into full memory exhaustion, causing OutOfMemoryError and denial of service in the JVM.

Root Cause

The root cause is a missing upper bound on the reassembly buffer relative to the peer's advertised maximum handshake message size. DTLSReliableHandshake had no maxHandshakeMessageSize parameter, so the constructor accepted any 24-bit length declared on the wire. This is a classic [CWE-789] pattern: allocation size is derived directly from untrusted input.

Attack Vector

The attack is network-based, requires no privileges, and requires no user interaction. An attacker sends a DTLS handshake fragment with a maliciously large 24-bit length field to any listener using Bouncy Castle's DTLSServerProtocol, or lures a Bouncy Castle client into a DTLS handshake against a hostile server that exploits the same code path in DTLSClientProtocol. The impact is confined to availability; confidentiality and integrity are not affected.

java

// Patch: bound the reassembly buffer by the peer's max handshake message size
// tls/src/main/java/org/bouncycastle/tls/DTLSReliableHandshake.java

private int next_send_seq = 0, next_receive_seq = 0;

private int maxHandshakeMessageSize;

DTLSReliableHandshake(TlsContext context, DTLSRecordLayer transport, int timeoutMillis, int initialResendMillis,
    DTLSRequest request, int maxHandshakeMessageSize)
{
    long currentTimeMillis = System.currentTimeMillis();

    this.recordLayer = transport;
    this.handshakeHash = new DeferredHash(context);
    this.handshakeTimeout = Timeout.forWaitMillis(timeoutMillis, currentTimeMillis);
    this.initialResendMillis = initialResendMillis;
    this.maxHandshakeMessageSize = maxHandshakeMessageSize;

    if (null != request)
    {
// Source: https://github.com/bcgit/bc-java/commit/2ea38942c7917f6d7ab4de93d8a5336d021df0d9

A follow-up commit hardened the floor value used by callers:

java

// tls/src/main/java/org/bouncycastle/tls/DTLSClientProtocol.java

DTLSReliableHandshake handshake = new DTLSReliableHandshake(clientContext, recordLayer,
    client.getHandshakeTimeoutMillis(), client.getHandshakeResendTimeMillis(), null,
    TlsUtils.getMaxHandshakeMessageSize(client));
// Source: https://github.com/bcgit/bc-java/commit/2d98721e71bbd822ffa0f84e088eea645cf679fa

Detection Methods for CVE-2026-59646

Indicators of Compromise

Java processes terminating with java.lang.OutOfMemoryError: Java heap space shortly after inbound UDP DTLS traffic
Sudden JVM heap growth correlated with connections on DTLS ports (commonly 4433, 5684, or application-specific UDP ports)
Repeated inbound DTLS ClientHello fragments from a small set of source addresses with unusually large declared length fields

Detection Strategies

Inventory applications for the vulnerable Bouncy Castle JARs (bcprov, bctls, bctls-fips) using software composition analysis or mvn dependency:tree
Inspect DTLS records at the perimeter for handshake fragments whose declared 24-bit length exceeds the negotiated or expected maximum handshake message size
Alert on Java service restarts or crash-loop patterns co-occurring with UDP traffic spikes on DTLS listeners

Monitoring Recommendations

Enable JVM heap and garbage collection telemetry, and forward JVM crash logs to a centralized log store for correlation
Monitor UDP packet rates and source diversity on DTLS-facing services; sudden narrow-source spikes are consistent with allocation-abuse attempts
Track library versions across the fleet and flag hosts still running Bouncy Castle Java below the fixed releases

How to Mitigate CVE-2026-59646

Immediate Actions Required

Upgrade Bouncy Castle for Java to 1.85 or later, LTS to 2.73.12 or later, and BC-FJA bctls-fips to 1.0.24, 2.0.24, or 2.1.24 depending on the series in use
Restrict network reachability of DTLS listeners to known peers via firewall or UDP allowlists until patching completes
Set an explicit low value for getMaxHandshakeMessageSize() on both client and server TlsPeer implementations

Patch Information

Upstream fixes are available in two commits: bc-java 2ea38942 adds a maxHandshakeMessageSize parameter to DTLSReliableHandshake, and bc-java 2d98721e routes callers through TlsUtils.getMaxHandshakeMessageSize() to enforce a consistent floor. See the Bouncy Castle CVE-2026-59646 advisory for release notes.

Workarounds

Override TlsPeer.getMaxHandshakeMessageSize() in application code to return a small bound (for example, 16384) consistent with expected certificate chain sizes
Terminate DTLS at a hardened gateway that enforces handshake size limits before traffic reaches Bouncy Castle
Apply per-source UDP rate limits on DTLS ports to reduce amplification from repeated malicious fragments

bash

# Verify installed Bouncy Castle versions in a Maven or Gradle project
mvn dependency:tree | grep -Ei 'bcprov|bctls|bcpkix'
./gradlew dependencies | grep -Ei 'bcprov|bctls|bcpkix'

# Confirm the fixed version is on the classpath
unzip -p bctls-jdk18on-1.85.jar META-INF/MANIFEST.MF | grep -i version