Skip to main content
Vulnerability Database/CVE-2026-63566

CVE-2026-63566: Bouncy Castle bc-csharp DTLS DOS Vulnerability

CVE-2026-63566 is a denial of service vulnerability in Bouncy Castle bc-csharp that enables remote attackers to exhaust memory through crafted DTLS handshake fragments. This article covers technical details, impact, and mitigation.

Published:

Disclaimer: This content was generated using AI. While we strive for accuracy, please verify critical information with official sources.

Sources: National Vulnerability Database, CISA KEV, Exploit-DB

CVE-2026-63566 Overview

CVE-2026-63566 is a denial-of-service vulnerability in the Legion of the Bouncy Castle bc-csharp library affecting versions prior to 2.7.0. The flaw resides in the Datagram Transport Layer Security (DTLS) handshake reassembly code (DtlsReliableHandshake, DtlsReassembler). A remote unauthenticated DTLS peer can send crafted handshake message fragments that force the library to allocate up to 16 MB of memory per pending message, across up to 16 pending messages per handshake, before the peer is authenticated. Both DTLS servers and DTLS clients are affected. TLS implementations using the same library are not affected.

Critical Impact

Unauthenticated remote attackers can exhaust server or client memory by sending a single crafted DTLS handshake fragment, causing denial of service across any bc-csharp DTLS endpoint before version 2.7.0.

Affected Products

  • Legion of the Bouncy Castle Inc. bc-csharp versions before 2.7.0
  • DTLS server implementations built on DtlsServerProtocol
  • DTLS client implementations built on DtlsClientProtocol

Discovery Timeline

  • 2026-10-02 - CVE-2026-63566 published to the National Vulnerability Database
  • 2026-10-02 - Last updated in NVD database

Technical Details for CVE-2026-63566

Vulnerability Analysis

The vulnerability is an uncontrolled memory allocation flaw classified under [CWE-789: Memory Allocation with Excessive Size Value]. During a DTLS handshake, incoming records can arrive as fragments that must be reassembled before the full handshake message is processed. In bc-csharp before 2.7.0, the reassembly buffer for each pending message was sized using the 24-bit length field declared in the fragment header. That field can declare up to 16 MB (2^24 bytes). The library allocated the full declared size on receipt of the first fragment, regardless of actual payload size.

A single empty fragment can therefore trigger a near-16 MB allocation. DTLS state permits up to 16 pending messages per handshake, so one attacker handshake can force roughly 256 MB of allocation before any cryptographic authentication occurs. Repeating this across many spoofed source addresses amplifies the memory pressure into a reliable denial-of-service condition.

Root Cause

The TLS code path in bc-csharp already enforced a peer-configurable MaxHandshakeMessageSize ceiling against the declared length. The equivalent check was missing from the DTLS reassembly path. The DtlsReliableHandshake constructor accepted no size bound, so DtlsReassembler trusted the attacker-controlled header length directly when allocating its backing buffer.

Attack Vector

The attack is remote and requires no authentication. An attacker sends a DTLS handshake fragment to a listening server, or responds to a client's ClientHello with a crafted fragment, declaring a 24-bit length near 0xFFFFFF and carrying no payload. The receiver immediately allocates the full declared buffer. Because DTLS is UDP-based, source addresses can be spoofed, making rate limiting and attribution more difficult than for TCP-based TLS.

The fix in commit 212d0647 introduces a maxHandshakeMessageSize parameter to DtlsReliableHandshake and enforces a 1024-byte floor:

csharp
 DtlsReliableHandshake handshake = new DtlsReliableHandshake(clientContext, recordLayer,
-    client.GetHandshakeTimeoutMillis(), TlsUtilities.GetHandshakeResendTimeMillis(client), null);
+    client.GetHandshakeTimeoutMillis(), TlsUtilities.GetHandshakeResendTimeMillis(client), null,
+    System.Math.Max(1024, client.GetMaxHandshakeMessageSize()));

Source: GitHub Commit 212d0647

A follow-up commit 99afef61 centralizes the floor calculation in TlsUtilities.GetMaxHandshakeMessageSize:

csharp
 DtlsReliableHandshake handshake = new DtlsReliableHandshake(serverContext, recordLayer,
     server.GetHandshakeTimeoutMillis(), TlsUtilities.GetHandshakeResendTimeMillis(server), request,
-    System.Math.Max(1024, server.GetMaxHandshakeMessageSize()));
+    TlsUtilities.GetMaxHandshakeMessageSize(server));

Source: GitHub Commit 99afef61

Detection Methods for CVE-2026-63566

Indicators of Compromise

  • Sudden spikes in managed heap allocation or working-set memory on processes hosting DTLS endpoints that use bc-csharp
  • UDP traffic to DTLS listeners containing handshake fragments with a 24-bit length field approaching 0xFFFFFF and small or zero payload
  • OutOfMemoryException or process termination events correlated with inbound DTLS handshake activity from unauthenticated peers

Detection Strategies

  • Inspect DTLS handshake records at the network layer and alert when the declared fragment length far exceeds the actual UDP payload size
  • Baseline normal DTLS handshake message sizes for your application and flag deviations exceeding the configured MaxHandshakeMessageSize
  • Correlate application-side memory pressure telemetry with inbound DTLS session attempts from new or high-volume source IPs

Monitoring Recommendations

  • Enable verbose logging on DTLS protocol handlers to capture the declared length field of inbound handshake fragments
  • Monitor .NET process metrics (GC heap, Gen2 collections, working set) on hosts terminating DTLS to catch allocation anomalies early
  • Track UDP flow records to DTLS ports and alert on bursts of short handshake packets from unique source addresses

How to Mitigate CVE-2026-63566

Immediate Actions Required

  • Upgrade bc-csharp to version 2.7.0 or later across all DTLS server and client deployments
  • Inventory applications that depend on Bouncy Castle for .NET, including transitive dependencies, and prioritize those exposing DTLS to untrusted networks
  • Override GetMaxHandshakeMessageSize in your TlsClient and TlsServer implementations to return a conservative value appropriate for your handshake profile

Patch Information

The vulnerability is fixed in bc-csharp 2.7.0. The remediation adds a maxHandshakeMessageSize parameter to DtlsReliableHandshake and enforces it against the declared 24-bit length during reassembly. See the GitHub CVE-2026-63566 Advisory and the fix commits 212d0647 and 99afef61.

Workarounds

  • Restrict DTLS endpoint exposure to trusted network segments using firewall or security group rules until the upgrade is deployed
  • Apply per-source UDP rate limiting on DTLS ports to reduce the amplification potential of unauthenticated handshake floods
  • Deploy memory and process watchdogs that restart affected services if resident memory exceeds expected handshake baselines
bash
# Upgrade bc-csharp to the fixed version using the .NET CLI
dotnet add package BouncyCastle.Cryptography --version 2.7.0

# Verify the installed package version
dotnet list package | grep -i bouncy

Disclaimer: This content was generated using AI. While we strive for accuracy, please verify critical information with official sources.

Sources: National Vulnerability Database, CISA KEV, Exploit-DB

Experience the Most Advanced Cybersecurity Platform

See how the world’s most intelligent, autonomous cybersecurity platform can protect your organization today and into the future.