CVE-2026-65178: NVIDIA NeMo RCE Vulnerability

CVE-2026-65178 Overview

NVIDIA NeMo contains a deserialization vulnerability in its dataset-loading workflow. A maliciously crafted model_config.yaml file can inject unsafe parameters processed during dataset initialization. Successful exploitation can lead to arbitrary code execution, data tampering, denial of service, and information disclosure on the host running NeMo.

The issue is tracked under CWE-502: Deserialization of Untrusted Data and requires local access with low privileges. No user interaction is required once the crafted configuration is loaded by a NeMo workflow.

Critical Impact

An attacker who supplies a malicious model_config.yaml to a NeMo pipeline can execute code in the context of the training or inference process, compromising confidentiality, integrity, and availability of the host.

Affected Products

• NVIDIA NeMo framework (dataset-loading workflow)
• Deployments consuming third-party or untrusted model_config.yaml files
• ML training and inference environments running affected NeMo versions

Discovery Timeline

• 2026-09-22 - CVE-2026-65178 published to NVD
• 2026-09-23 - Last updated in NVD database
• Vendor advisory - Published in the NVIDIA product-security repository

Technical Details for CVE-2026-65178

Vulnerability Analysis

The vulnerability resides in how NeMo processes model_config.yaml when loading datasets. The configuration parser accepts parameters that influence object instantiation and callable resolution during dataset construction. When the YAML file contains attacker-controlled fields, the loader can materialize objects or invoke functions that were never intended to be reachable from configuration.

Because dataset loading occurs early in the training and inference pipeline, injected parameters execute with the same privileges as the NeMo process. This enables arbitrary code execution, tampering with model artifacts, exhaustion of local resources, or disclosure of environment secrets and dataset contents.

Exploitation requires local access with low privileges and no user interaction. The impact on confidentiality, integrity, and availability is high because the NeMo process typically holds access to training data, checkpoints, credentials for storage backends, and GPU resources.

Root Cause

The root cause is unsafe deserialization of untrusted configuration data [CWE-502]. The dataset-loading logic trusts fields inside model_config.yaml to specify class names, callables, or constructor arguments without adequately restricting which types can be instantiated. This pattern is common in ML frameworks that use dynamic class resolution to keep pipelines flexible.

Attack Vector

An attacker delivers a crafted model_config.yaml to a target user or system. Delivery paths include shared model repositories, community model hubs, git repositories imported into training pipelines, and internal file shares. When a NeMo user runs a training or evaluation workflow referencing the file, the malicious parameters are deserialized and executed in-process.

The vulnerability manifests when configuration parsing resolves class or function references from YAML values. See the NVIDIA security advisory for vendor-provided technical details and affected versions.

Detection Methods for CVE-2026-65178

Indicators of Compromise

• Unexpected child processes spawned by Python interpreters running NeMo workflows
• model_config.yaml files referencing unfamiliar Python modules, callables, or !!python/object style tags
• Outbound network connections from training hosts to unknown destinations during dataset initialization
• New or modified files in NeMo working directories that were not produced by legitimate training runs

Detection Strategies

• Scan repositories and shared storage for YAML configurations containing dynamic class instantiation directives before they reach training hosts
• Baseline expected process trees for NeMo jobs and alert on deviations during dataset loading
• Monitor for anomalous file system, network, or credential access originating from ML pipeline processes

Monitoring Recommendations

• Log all model_config.yaml sources, hashes, and provenance for every training run
• Forward endpoint and host telemetry from GPU worker nodes to a central analytics platform for retrospective hunting
• Alert on invocations of subprocess, os.system, or unusual importlib activity within NeMo processes

How to Mitigate CVE-2026-65178

Immediate Actions Required

• Upgrade NeMo to the fixed version listed in the NVIDIA product-security advisory
• Treat all externally sourced model_config.yaml files as untrusted and review them before use
• Restrict who can write to configuration paths consumed by NeMo pipelines
• Run NeMo workflows under least-privilege service accounts isolated from production credentials

Patch Information

NVIDIA has published remediation guidance in the product-security repository entry for bulletin 5885. Apply the vendor-supplied fixed release for NeMo and rebuild any container images that embed vulnerable versions. Refer to the NVD entry for CVE-2026-65178 for the authoritative reference list.

Workarounds

• Load only model_config.yaml files from trusted, version-controlled sources with signature verification
• Execute NeMo dataset loading inside sandboxed containers with restricted network egress and no access to sensitive secrets
• Use YAML safe-loading configurations and remove custom tag resolvers where possible
• Implement code review and static analysis for configuration files entering the ML supply chain

# Configuration example: run NeMo in a restricted container
docker run --rm \
  --read-only \
  --cap-drop=ALL \
  --network=none \
  --user 10001:10001 \
  -v /data/models:/models:ro \
  nvcr.io/nvidia/nemo:<patched-version> \
  python train.py --config /models/reviewed/model_config.yaml