CVE-2024-12044: MMDetection RCE Vulnerability

CVE-2024-12044 Overview

A remote code execution vulnerability exists in open-mmlab/mmdetection version v3.3.0. The vulnerability is due to the use of the pickle.loads() function in the all_reduce_dict() distributed training API without proper sanitization. This allows an attacker to execute arbitrary code by broadcasting a malicious payload to the distributed training network.

Critical Impact

Attackers can achieve remote code execution on any node participating in MMDetection's distributed training by exploiting insecure deserialization of pickled data, potentially compromising entire machine learning training clusters.

Affected Products

• open-mmlab/mmdetection v3.3.0
• Distributed training deployments using the all_reduce_dict() API

Discovery Timeline

• 2025-03-20 - CVE CVE-2024-12044 published to NVD
• 2025-03-20 - Last updated in NVD database

Technical Details for CVE-2024-12044

Vulnerability Analysis

This vulnerability is classified as CWE-502: Deserialization of Untrusted Data. The core issue lies in MMDetection's distributed training infrastructure, specifically within the all_reduce_dict() function. This function is designed to synchronize dictionary data across multiple nodes in a distributed training environment.

The vulnerability allows an attacker with access to the distributed training network to execute arbitrary code on any participating node. Since distributed training environments often consist of multiple high-performance GPU servers, successful exploitation could lead to complete compromise of an entire machine learning training cluster, including access to proprietary training data, model weights, and computational resources.

Root Cause

The root cause of this vulnerability is the unsafe use of Python's pickle.loads() function to deserialize data received over the network without any validation or sanitization. Python's pickle module is inherently unsafe when processing untrusted data because it can execute arbitrary Python code during deserialization.

When data is broadcast across the distributed training network using the all_reduce_dict() API, the receiving nodes deserialize the incoming payload using pickle.loads(). An attacker who can inject malicious data into this communication channel can craft a specially designed pickle payload that executes arbitrary code when deserialized.

Attack Vector

The attack is network-based and requires the attacker to have access to the distributed training network. The attacker can broadcast a malicious pickle payload to the distributed training network, which will be deserialized by all participating nodes. Since distributed training environments often use shared networks and may not implement strong inter-node authentication, an attacker positioned on the same network segment could potentially inject malicious payloads.

The exploitation does not require any privileges on the target system or user interaction, making this vulnerability particularly dangerous in environments where network isolation is not strictly enforced. Successful exploitation results in arbitrary code execution with the privileges of the training process, typically providing full access to the training data, models, and system resources.

For technical details on this vulnerability and proof-of-concept information, see the Huntr Bug Bounty Listing.

Detection Methods for CVE-2024-12044

Indicators of Compromise

• Unexpected network connections or data transfers within the distributed training network from unknown sources
• Unusual processes spawned by the Python training process, especially shell commands or network utilities
• Abnormal system behavior on training nodes such as unexpected file modifications or new scheduled tasks
• Suspicious pickle payloads containing Python bytecode or unusual object types in network traffic

Detection Strategies

• Implement network monitoring to detect anomalous traffic patterns within distributed training clusters
• Deploy endpoint detection and response (EDR) solutions like SentinelOne on all training nodes to detect post-exploitation behavior
• Monitor for suspicious child processes spawned by Python training scripts, particularly shell executions
• Implement application-level logging for the all_reduce_dict() function to capture deserialization events

Monitoring Recommendations

• Enable comprehensive logging for distributed training communications and inter-node data exchanges
• Configure alerts for any unauthorized network connections to or from training infrastructure
• Monitor for file system modifications in training environments that occur outside of expected model checkpoint saves
• Implement runtime application self-protection (RASP) or code-level monitoring for pickle deserialization operations

How to Mitigate CVE-2024-12044

Immediate Actions Required

• Audit your environment to determine if you are running open-mmlab/mmdetection v3.3.0 with distributed training enabled
• Isolate distributed training networks from untrusted network segments and implement strict network segmentation
• Review access controls for who can connect to and communicate with distributed training nodes
• Consider temporarily disabling distributed training if the network cannot be adequately secured until a patch is available

Patch Information

Check the Huntr Bug Bounty Listing and the open-mmlab/mmdetection GitHub repository for updates on official patches. Monitor for new releases of MMDetection that address this vulnerability. When a patched version becomes available, update all distributed training nodes immediately.

Workarounds

• Implement strong network isolation for distributed training clusters, ensuring only trusted nodes can participate
• Replace pickle.loads() with safer serialization alternatives such as JSON or MessagePack where possible in custom code
• Implement input validation and authentication for inter-node communications in the distributed training pipeline
• Deploy SentinelOne Singularity Platform on all training nodes to detect and prevent exploitation attempts through behavioral AI analysis

# Network isolation example using iptables
# Restrict distributed training communication to known trusted nodes only
iptables -A INPUT -p tcp --dport 29500 -s 10.0.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 29500 -j DROP