CVE-2026-46344: Openquantumsafe Liboqs DOS Vulnerability

CVE-2026-46344 Overview

CVE-2026-46344 is an out-of-bounds read vulnerability in liboqs, the Open Quantum Safe C-language cryptographic library implementing post-quantum cryptography algorithms. The flaw affects the stateful signature verification code for XMSS (eXtended Merkle Signature Scheme) and XMSS^MT (Multi-Tree XMSS). When xmss_sign_open or xmssmt_sign_open is invoked with a public key whose OID bytes reference a parameter set with a larger sig_bytes than the caller-supplied signature buffer, the implementation reads past the end of that buffer. The out-of-bounds bytes feed an internal hash computation and are not returned to the caller, eliminating any data-leakage oracle. The vulnerability is fixed in liboqs version 0.16.0.

Critical Impact

A remote attacker can trigger a process crash in any service performing XMSS or XMSS^MT signature verification with liboqs versions prior to 0.16.0, resulting in denial of service.

Affected Products

• Open Quantum Safe liboqs versions prior to 0.16.0
• Applications and libraries embedding liboqs for XMSS stateful signature verification
• Downstream integrations using xmss_sign_open or xmssmt_sign_open APIs

Discovery Timeline

• 2026-05-29 - CVE-2026-46344 published to NVD
• 2026-06-04 - Last updated in NVD database

Technical Details for CVE-2026-46344

Vulnerability Analysis

The vulnerability is an out-of-bounds read [CWE-125] in the XMSS and XMSS^MT stateful signature verification paths inside liboqs. XMSS is a hash-based, stateful post-quantum signature scheme standardized in RFC 8391, with multiple parameter sets that produce signatures of differing fixed sizes. The verification functions xmss_sign_open and xmssmt_sign_open derive the expected signature length (sig_bytes) from the algorithm identifier encoded in the first four bytes of the public key (pk[0..3]).

The defect occurs because the calling layer sizes the signature buffer based on the algorithm the caller selected, while the inner verification logic re-parses the OID from the supplied public key and uses that OID's sig_bytes. When the OID embedded in the public key references a parameter set with a larger signature size than the algorithm chosen at the API level, the verifier indexes bytes beyond the end of the signature buffer.

The out-of-bounds bytes are consumed only by an internal hash computation and are never returned to the caller. The advisory explicitly notes the absence of a confidentiality oracle, mirroring the prior CVE-2026-44518. The observable effect is a crash of the verifying process when the read crosses into an unmapped page, producing a network-reachable denial of service.

Root Cause

The root cause is inconsistent trust placement on the algorithm OID. The API-level signature buffer is sized using the caller-declared algorithm, but the internal XMSS routines re-derive sig_bytes from pk[0..3] without validating that the embedded OID matches the declared algorithm. This mismatch lets a crafted public key dictate buffer indexing beyond allocated bounds.

Attack Vector

An attacker who can supply or influence the public key and signature passed to a verification function can trigger the out-of-bounds read. The attack vector is network-reachable when liboqs is used in protocols or services that accept attacker-controlled XMSS public keys. No authentication or user interaction is required. The vulnerability affects availability only; integrity and confidentiality are unaffected because the OOB data is hashed and the verification cryptographically fails.

The vulnerability manifests inside xmss_sign_open and xmssmt_sign_open when the embedded OID in the public key references a parameter set with a larger sig_bytes than the buffer the caller passed in. See the GitHub Security Advisory and the upstream fix commit for the verified patch details.

Detection Methods for CVE-2026-46344

Indicators of Compromise

• Unexpected segmentation faults or SIGSEGV terminations in processes linking liboqs during XMSS or XMSS^MT verification operations.
• Repeated crash-restart cycles in TLS, VPN, or PKI services performing post-quantum signature verification.
• Inbound network sessions delivering XMSS public keys whose first four OID bytes do not match the algorithm negotiated at the protocol layer.

Detection Strategies

• Inventory all binaries and containers linking liboqs and flag versions earlier than 0.16.0.
• Inspect crash dumps from services using liboqs for stack frames inside xmss_sign_open or xmssmt_sign_open.
• Validate at the application layer that the OID parsed from pk[0..3] matches the algorithm identifier passed to the verification API before invoking liboqs.

Monitoring Recommendations

• Alert on abnormal process termination rates for services that perform post-quantum signature verification.
• Track Software Bill of Materials (SBOM) data for the liboqs component version across production workloads.
• Forward crash telemetry and core dump metadata to a centralized analytics pipeline to correlate denial-of-service patterns against external traffic.

How to Mitigate CVE-2026-46344

Immediate Actions Required

• Upgrade liboqs to version 0.16.0 or later across all systems and rebuild dependent applications and containers.
• Audit downstream packages and language bindings (for example liboqs-python, oqs-provider) and update them to releases that link the patched library.
• Restrict exposure of XMSS and XMSS^MT verification endpoints to trusted peers until patched builds are deployed.

Patch Information

The fix is delivered in liboqs0.16.0. The upstream patch is recorded in commit 077e32a94f39af02209dbbc680bf8a43b774b305 and described in the Open Quantum Safe security advisory GHSA-2wxh-55qf-c7wg. The patch reconciles the algorithm OID parsed from the public key with the caller-declared algorithm before indexing the signature buffer.

Workarounds

• Validate that pk[0..3] matches the expected algorithm OID at the application layer before calling xmss_sign_open or xmssmt_sign_open.
• Reject XMSS and XMSS^MT public keys originating from untrusted sources until the patched version is in place.
• Disable post-quantum stateful signature verification code paths in services where they are not strictly required.

# Configuration example: build and install the patched liboqs release
git clone --branch 0.16.0 https://github.com/open-quantum-safe/liboqs.git
cd liboqs
mkdir build && cd build
cmake -GNinja -DCMAKE_INSTALL_PREFIX=/usr/local ..
ninja
sudo ninja install

# Verify the installed version
pkg-config --modversion liboqs