CVE-2026-12852 Overview
CVE-2026-12852 is a memory allocation flaw in Bouncy Castle for Java versions prior to 1.85. The Messaging Layer Security (MLS) wire decoder allocates a buffer sized from an attacker-declared opaque length field before validating that the input actually contains that many bytes. Because opaque fields appear in the first bytes of unauthenticated MLS structures such as PublicMessage, KeyPackage, Welcome, and PrivateMessage, a few-byte message can force a heap allocation of up to ~1 GiB before any signature or MAC verification. The weakness maps to [CWE-789: Memory Allocation with Excessive Size Value].
Critical Impact
A remote unauthenticated attacker can trigger large heap allocations with tiny wire messages, exhausting memory and causing denial of service against any Java service that decodes MLS traffic using Bouncy Castle.
Affected Products
- Bouncy Castle for Java versions before 1.85
- Java applications that consume MLS messages via MLSInputStream
- Services exposing MLS endpoints (PublicMessage, KeyPackage, Welcome, PrivateMessage handlers)
Discovery Timeline
- 2026-08-03 - CVE-2026-12852 published to NVD
- 2026-08-04 - Last updated in NVD database
Technical Details for CVE-2026-12852
Vulnerability Analysis
The MLS wire decoder in Bouncy Castle reads a Varint-encoded length prefix from incoming messages and passes that length directly to new byte[size]. The decoder performs no sanity check against the number of bytes actually remaining in the input stream. An attacker can encode a length of up to 0x3FFFFFFF (roughly 1 GiB) in a message that is only a few bytes long. The Java virtual machine will then attempt to allocate the declared buffer, consuming heap memory disproportionate to the network cost of sending the message.
Because opaque fields sit ahead of any cryptographic authentication data, the allocation happens pre-auth. A modest volume of concurrent malformed messages is sufficient to exhaust the heap of the target process, producing OutOfMemoryError conditions and denial of service.
Root Cause
The readAll(int size) helper inside MLSInputStream trusted the length declared in the wire format. The existing bounds check present in slice() was not mirrored in the buffer-allocation path, so a length value that exceeded available() still drove the allocation.
Attack Vector
Exploitation requires only network access to an application that decodes MLS wire messages using vulnerable Bouncy Castle releases. No authentication, no user interaction, and no session state are required. The attacker sends a small crafted message containing an oversized opaque-field Varint length.
byte[] readAll(int size)
throws IOException
{
+ // Validate the requested length against the bytes actually remaining before allocating, so
+ // an attacker-declared opaque/Varint length (up to ~1 GiB) in a tiny unauthenticated wire
+ // message cannot drive a huge new byte[] allocation and exhaust the heap. available() is
+ // exact for the backing ByteArrayInputStream.
+ if (size < 0 || size > available())
+ {
+ throw new IOException("Attempt to read beyond end of buffer");
+ }
byte[] data = new byte[size];
if (size == 0)
{
Source: Bouncy Castle commit a747038 — this patch adds the pre-allocation bounds check that closes the flaw.
Detection Methods for CVE-2026-12852
Indicators of Compromise
- Java process OutOfMemoryError or garbage-collection thrashing correlated with inbound MLS traffic
- Sudden heap growth on services that expose MLS endpoints, without a matching increase in message throughput
- Small inbound packets followed by large byte[] allocations visible in JVM allocation profiling
Detection Strategies
- Enumerate Java dependencies with a Software Bill of Materials (SBOM) tool and flag any bcprov, bctls, or bcmls artifacts below version 1.85
- Inspect MLS message parsers for Varint length values that exceed the byte count of the enclosing frame
- Add JVM heap and garbage-collection telemetry to SIEM pipelines and alert on rapid heap growth on MLS-handling services
Monitoring Recommendations
- Monitor Java Flight Recorder or JMX metrics for java.lang.OutOfMemoryError events on services processing MLS traffic
- Track connection patterns to MLS endpoints, alerting on repeated undersized messages that trigger parser exceptions
- Correlate IOException: Attempt to read beyond end of buffer log entries after patching to confirm attack attempts are being rejected
How to Mitigate CVE-2026-12852
Immediate Actions Required
- Upgrade Bouncy Castle for Java to version 1.85 or later across all applications and container images
- Audit transitive dependencies with mvn dependency:tree or gradle dependencies to catch bundled copies of vulnerable Bouncy Castle jars
- Restart affected JVMs after upgrade to ensure the older classes are unloaded from long-running processes
Patch Information
The fix landed in commit a747038bb5bbd5e29fb2b7607ab38af1fd8d1790 and shipped in Bouncy Castle for Java 1.85. The patched readAll method validates the requested size against available() before allocating, throwing an IOException when the declared length exceeds the bytes remaining. See the CVE-2026-12852 wiki entry for the vendor's technical description.
Workarounds
- Place MLS-handling services behind a reverse proxy or message broker that enforces a strict maximum frame size well below 1 GiB
- Constrain JVM heap size with -Xmx so a single oversized allocation fails fast rather than degrading the entire host
- Rate-limit unauthenticated inbound MLS messages per source address to blunt concurrent allocation attempts
# Verify the installed Bouncy Castle version and fail the build on vulnerable releases
mvn dependency:tree -Dincludes=org.bouncycastle | grep -E 'bcprov|bctls|bcmls'
# Example Gradle constraint pinning the fixed version
# build.gradle
dependencies {
constraints {
implementation('org.bouncycastle:bcprov-jdk18on:1.85') {
because 'CVE-2026-12852 MLS wire decoder unbounded allocation'
}
}
}
Disclaimer: This content was generated using AI. While we strive for accuracy, please verify critical information with official sources.
