CVE-2026-85670: Hugging Face Tokenizers DOS Vulnerability

CVE-2026-85670 Overview

CVE-2026-85670 is an out-of-bounds buffer access vulnerability in the Hugging Face tokenizers Rust library, specifically in BpeBuilder::build located at tokenizers/src/models/bpe/model.rs. The flaw triggers when a tokenizer loads a crafted tokenizer.json via Tokenizer::from_file or Tokenizer::from_str. A merge rule whose concatenated token exceeds the longest vocabulary key overruns a scratch buffer, causing Rust to panic and abort the host process. The defect affects both native Rust callers and applications using the library through Foreign Function Interface (FFI) bindings such as Python. Version 0.23.1 is confirmed affected.

Critical Impact
A crafted tokenizer.json file causes denial of service at load time in any application that consumes untrusted tokenizer assets, including machine learning inference services.

Affected Products

Hugging Face tokenizers Rust crate, version 0.23.1
Python bindings and other FFI embeddings that wrap the affected crate
Downstream ML applications that load third-party tokenizer.json files

Discovery Timeline

2026-09-04 - CVE-2026-85670 published to the National Vulnerability Database (NVD)
2026-09-04 - Last updated in NVD database

Technical Details for CVE-2026-85670

Vulnerability Analysis

The vulnerability is classified as an out-of-bounds write [CWE-787] in the Byte Pair Encoding (BPE) model builder. During construction, BpeBuilder::build allocates a scratch buffer sized to the longest key present in the vocabulary. The builder then iterates over merge rules and writes each concatenated merge token into that buffer. The code assumes concatenated merge tokens never exceed the longest vocabulary key, but this invariant is never enforced against attacker-supplied input.

When a merge rule concatenates two tokens whose combined length is greater than the scratch buffer, the write overruns bounds. Rust's runtime detects the invalid access and panics, which aborts the process. In FFI embeddings such as the Python bindings, this aborts the host interpreter or worker instead of raising a catchable exception.

A secondary defect at the same location causes a usize underflow when continuing_subword_prefix is configured and a merge token is shorter than the prefix length. In debug builds this panics. In release builds the underflow can lead to memory corruption because the wrapped size drives subsequent buffer operations.

Root Cause

The root cause is missing validation of relationships between attacker-controlled fields in the tokenizer configuration. The builder trusts that the maximum vocabulary key length is a sound upper bound for concatenated merge tokens and that continuing_subword_prefix is never longer than any merge token. Neither invariant is enforced before writes occur.

Attack Vector

Exploitation requires an application to load a malicious tokenizer.json through Tokenizer::from_file or Tokenizer::from_str. No encoding or inference call is required; the fault fires during model load. Services that automatically fetch tokenizers from third-party repositories or accept user-uploaded model artifacts are exposed. The user interaction requirement reflects the need to load the attacker-supplied file.

The vulnerability mechanism is documented in the VulnCheck Advisory on BPEBuilder Buffer Overflow and tracked upstream in GitHub Tokenizers Issue #2094. The affected source is available in the BPE Model Source Code.

Detection Methods for CVE-2026-85670

Indicators of Compromise

Unexpected process aborts or SIGABRT signals in ML inference workers immediately after loading a tokenizer
Rust panic messages referencing bpe/model.rs or BpeBuilder::build in application logs
tokenizer.json files sourced from untrusted registries containing merge entries longer than any vocabulary key
Repeated container or pod restarts correlated with model or tokenizer refresh events

Detection Strategies

Inspect tokenizer.json files before load and flag any merge whose concatenated length exceeds the maximum vocab key length
Flag any tokenizer.json where continuing_subword_prefix is longer than the shortest merge token
Monitor inference services for panic-induced process termination correlated with model-load events
Enumerate deployed versions of the tokenizers crate and Python tokenizers package to identify hosts running 0.23.1 or earlier

Monitoring Recommendations

Forward stderr and application logs from ML worker processes to centralized logging and alert on Rust panic strings
Track model and tokenizer download events with content hashes to establish provenance
Alert on abnormal restart rates for inference containers that consume externally sourced artifacts

How to Mitigate CVE-2026-85670

Immediate Actions Required

Upgrade the tokenizers crate and Python package beyond the affected 0.23.1 release to a fixed version
Restrict tokenizer sources to trusted internal registries and pin artifacts by cryptographic hash
Validate any externally sourced tokenizer.json for merge-token and vocabulary-key length invariants before loading
Isolate tokenizer load operations in sandboxed workers so a panic does not terminate a shared inference process

Patch Information

The upstream fix is present in the tokenizers repository following the affected 0.23.1 release. Review the Hugging Face tokenizers repository and the v0.23.2 BPE model source to confirm the fixed version pinned in your environment. Rebuild and redeploy any container images, Python virtual environments, or Rust binaries that statically embed the crate.

Workarounds

Block loading of tokenizer.json files that originate outside of an approved allowlist
Add a preflight parser that rejects tokenizer configurations violating the length invariants abused by this bug
Run tokenizer loading in a supervised subprocess so a panic aborts only that worker and can be restarted safely
Disable automatic pulls of tokenizer assets from public model hubs in production inference pipelines

bash

# Example: pin a fixed tokenizers version in a Python requirements file
# Replace <fixed-version> with a release later than 0.23.1 that contains the patch
pip install 'tokenizers>=<fixed-version>'

# Example: pin the Rust crate in Cargo.toml
# tokenizers = ">=<fixed-version>"