CVE-2025-23348: Nvidia Megatron-lm RCE Vulnerability

CVE-2025-23348 Overview

CVE-2025-23348 is a code injection vulnerability [CWE-94] in NVIDIA Megatron-LM, a large language model training framework. The flaw resides in the pretrain_gpt script, where maliciously crafted input data can trigger arbitrary code execution during model pretraining. A local attacker with low privileges can exploit this issue without user interaction. Successful exploitation leads to code execution, privilege escalation, information disclosure, and data tampering within the training environment.

Critical Impact
A local, low-privileged attacker can supply malicious training data to the pretrain_gpt script and gain code execution with the privileges of the training process, compromising GPU workloads and sensitive model data.

Affected Products

NVIDIA Megatron-LM versions prior to the patched release
NVIDIA Megatron-LM 0.13.0 (including release candidates rc0 through rc4)
All platforms running vulnerable Megatron-LM builds

Discovery Timeline

2025-09-24 - CVE-2025-23348 published to the National Vulnerability Database (NVD)
2026-06-17 - Last updated in the NVD database

Technical Details for CVE-2025-23348

Vulnerability Analysis

The vulnerability exists in the pretrain_gpt script distributed with NVIDIA Megatron-LM. The script processes attacker-controlled input data during model pretraining and fails to safely handle content that can influence code paths executed by the Python runtime. When malicious data is ingested, it triggers code injection during script execution.

Megatron-LM is widely deployed on multi-GPU systems used to train large language models. Because the pretraining process typically runs with access to training corpora, checkpoints, and GPU resources, successful exploitation grants the attacker the same access. The impact covers confidentiality, integrity, and availability of the training environment.

Root Cause

The root cause is improper control of code generation, mapped to [CWE-94]. The pretrain_gpt script accepts input data that flows into a code-executing sink without adequate sanitization or validation. When an attacker controls that data, the script executes attacker-supplied logic instead of treating the input strictly as data.

Attack Vector

Exploitation requires local access with low privileges and no user interaction. An attacker who can stage a malicious dataset, configuration, or auxiliary file consumed by pretrain_gpt can trigger code execution when the script runs. Shared training clusters, multi-tenant GPU environments, and CI/CD pipelines that ingest untrusted datasets are the most exposed scenarios. See the NVIDIA security bulletin for vendor details.

// No verified public exploit code is available for CVE-2025-23348.
// Refer to the NVIDIA advisory and NVD entry for authoritative technical details.

Detection Methods for CVE-2025-23348

Indicators of Compromise

Unexpected child processes spawned by python interpreters running pretrain_gpt or related Megatron-LM entry points.
Outbound network connections initiated from training hosts during dataset loading or preprocessing phases.
Modifications to model checkpoints, tokenizer files, or dataset artifacts outside of authorized workflows.
New or altered files in Megatron-LM working directories that do not match expected training outputs.

Detection Strategies

Monitor process trees for pretrain_gpt.py invocations that launch shells, package managers, or interpreters not required for training.
Alert on Megatron-LM processes making unexpected outbound connections or accessing credential stores.
Baseline dataset and checkpoint file hashes and flag deviations before and after training runs.

Monitoring Recommendations

Enable command-line and process auditing on GPU training hosts and forward logs to a centralized analytics platform.
Track user activity around dataset ingestion, including who staged files and from which source.
Correlate Megatron-LM job execution with file, process, and network telemetry to identify anomalies during pretraining.

How to Mitigate CVE-2025-23348

Immediate Actions Required

Upgrade NVIDIA Megatron-LM to the fixed release identified in the NVIDIA Support Response.
Restrict local access to training hosts to trusted operators and enforce least privilege for accounts that run pretrain_gpt.
Audit existing datasets, tokenizer artifacts, and configuration files for untrusted or unverified content.
Isolate training workloads that must consume third-party datasets into dedicated, sandboxed environments.

Patch Information

NVIDIA has released updated Megatron-LM builds that address CVE-2025-23348. Consult the NVIDIA security bulletin for fixed version numbers and download instructions. Additional technical metadata is available on the NVD CVE-2025-23348 Detail page and the CVE.org record.

Workarounds

Restrict pretrain_gpt execution to curated datasets that have been reviewed and hashed before use.
Run Megatron-LM training jobs inside containers or virtual machines with limited filesystem, network, and credential access.
Enforce mandatory access control policies (SELinux, AppArmor) on training hosts to constrain the blast radius of code injection.

bash

# Example: run pretrain_gpt in an isolated container with a read-only, verified dataset mount
docker run --rm \
  --gpus all \
  --network none \
  --read-only \
  -v /srv/datasets/verified:/data:ro \
  -v /srv/megatron/workspace:/workspace \
  --user 1000:1000 \
  nvcr.io/nvidia/megatron-lm:<patched-version> \
  python pretrain_gpt.py --data-path /data/corpus --save /workspace/checkpoints