CVE-2023-49935: SchedMD Slurm Auth Bypass Vulnerability

CVE-2023-49935 Overview

An Incorrect Access Control vulnerability has been discovered in SchedMD Slurm versions 23.02.x and 23.11.x that allows attackers to bypass message integrity protections in the slurmd process. The vulnerability enables the reuse of root-level authentication tokens during RPC interactions, effectively circumventing the MUNGE credential hashing mechanisms designed to prevent unauthorized credential reuse.

Critical Impact

Attackers with low-privilege network access can reuse root-level authentication tokens to bypass security controls, potentially gaining unauthorized access to HPC cluster resources and executing privileged operations.

Affected Products

• SchedMD Slurm 23.02.x (versions prior to 23.02.7)
• SchedMD Slurm 23.11 (including 23.11-rc1, versions prior to 23.11.1)
• Systems using MUNGE authentication with slurmd process

Discovery Timeline

• 2023-12-14 - CVE-2023-49935 published to NVD
• 2025-11-04 - Last updated in NVD database

Technical Details for CVE-2023-49935

Vulnerability Analysis

This vulnerability represents a significant authentication bypass flaw in the Slurm workload manager's daemon communication layer. The slurmd process, which runs on compute nodes and manages job execution, fails to properly validate the integrity of incoming RPC messages. Specifically, the vulnerability allows an attacker to capture and replay root-level MUNGE authentication tokens.

MUNGE (MUNGE Uid 'N' Gid Emporium) is the authentication mechanism used by Slurm to verify user credentials across the cluster. Under normal operation, RPC message hashes prevent the reuse of authentication credentials. However, the flawed implementation in affected versions allows attackers to bypass these hash-based protections, enabling credential replay attacks.

The vulnerability is classified under CWE-613 (Insufficient Session Expiration), indicating that the authentication tokens remain valid beyond their intended lifecycle, allowing malicious reuse.

Root Cause

The root cause stems from inadequate message integrity verification in the slurmd RPC handling code. The hashing mechanism designed to protect against MUNGE credential reuse contains a bypass condition that allows authenticated tokens to be replayed. This enables an attacker who has obtained or intercepted a root-level authentication token to reuse it for subsequent unauthorized operations against the slurmd process.

Attack Vector

The attack can be executed remotely over the network by an attacker with low-privilege access to the Slurm cluster. The exploitation flow involves:

1. The attacker obtains or intercepts a valid root-level MUNGE credential token from legitimate cluster communications
2. The attacker crafts malicious RPC messages reusing the captured authentication token
3. Due to the message integrity bypass, slurmd accepts the replayed credentials as valid
4. The attacker gains unauthorized root-level access to execute privileged operations on compute nodes

The vulnerability does not require user interaction and has low attack complexity, making it particularly dangerous in shared HPC environments where multiple users have network access to cluster infrastructure.

Detection Methods for CVE-2023-49935

Indicators of Compromise

• Unusual patterns of repeated MUNGE credential usage from unexpected source IPs
• Anomalous slurmd RPC activity originating from non-controller nodes
• Unexpected privilege escalation events on compute nodes
• Multiple authentication attempts using identical credential tokens within short time windows

Detection Strategies

• Monitor slurmd logs for duplicate or replayed authentication token patterns
• Implement network traffic analysis to detect suspicious RPC message patterns between cluster nodes
• Deploy file integrity monitoring on Slurm configuration and binary files
• Enable verbose logging on slurmd processes to capture detailed authentication events

Monitoring Recommendations

• Configure alerting for authentication anomalies in Slurm audit logs
• Monitor network connections to slurmd ports (typically 6818) for unusual traffic patterns
• Implement correlation rules to detect credential reuse across multiple nodes
• Review slurmctld and slurmd logs regularly for failed or suspicious authentication attempts

How to Mitigate CVE-2023-49935

Immediate Actions Required

• Upgrade SchedMD Slurm to version 23.02.7 or 23.11.1 immediately
• Audit recent slurmd authentication logs for signs of credential reuse attacks
• Review cluster access controls and restrict network access to slurmd ports
• Rotate MUNGE keys after patching to invalidate any potentially captured credentials

Patch Information

SchedMD has released security patches addressing this vulnerability. The fixed versions are 23.02.7 for the 23.02 series and 23.11.1 for the 23.11 series. Administrators should consult the Slurm Security Announcement and SchedMD Security Archive for detailed upgrade instructions.

Fedora users should check the Fedora Package Announcements for distribution-specific updates.

Workarounds

• Implement network segmentation to restrict access to slurmd communication ports to authorized management nodes only
• Enable additional authentication logging to detect potential exploitation attempts
• Consider implementing IP-based access controls on compute nodes to limit RPC communication sources
• Deploy intrusion detection rules to monitor for credential replay patterns

# Configuration example - Restrict slurmd port access via iptables
# Allow only slurmctld server to communicate with slurmd
iptables -A INPUT -p tcp --dport 6818 -s <slurmctld_ip> -j ACCEPT
iptables -A INPUT -p tcp --dport 6818 -j DROP

# Regenerate MUNGE key after patching (run on all nodes)
# mungekey -f -k /etc/munge/munge.key
# systemctl restart munge