CVE-2023-41914: Schedmd Slurm Race Condition Vulnerability

CVE-2023-41914 Overview

CVE-2023-41914 is a high-severity race condition vulnerability affecting SchedMD Slurm, a widely deployed open-source workload manager used in high-performance computing (HPC) clusters and supercomputing environments. The vulnerability exists in Slurm versions 23.02.x before 23.02.6 and 22.05.x before 22.05.10, allowing attackers to exploit filesystem race conditions to gain ownership of files, overwrite existing files, or delete files from the system.

Critical Impact

Local attackers with low privileges can exploit this race condition to escalate privileges, corrupt critical system files, or cause denial of service in HPC cluster environments.

Affected Products

• SchedMD Slurm 23.02.x (versions before 23.02.6)
• SchedMD Slurm 22.05.x (versions before 22.05.10)
• Fedora 39 (bundled Slurm packages)

Discovery Timeline

• 2023-11-03 - CVE-2023-41914 published to NVD
• 2024-11-21 - Last updated in NVD database

Technical Details for CVE-2023-41914

Vulnerability Analysis

This vulnerability is classified as CWE-362 (Concurrent Execution using Shared Resource with Improper Synchronization), commonly known as a race condition. The flaw specifically manifests as a Time-of-Check Time-of-Use (TOCTOU) vulnerability in Slurm's file handling operations.

In the context of Slurm's architecture, file operations are performed during job scheduling, resource allocation, and logging activities. The race condition occurs when there is a window of time between when a file's properties are checked (such as ownership or permissions) and when the file is actually used or modified. An attacker who can predict or influence file operations can exploit this timing gap to manipulate files in unauthorized ways.

Root Cause

The root cause of CVE-2023-41914 lies in improper synchronization during filesystem operations within Slurm's codebase. When Slurm performs file operations, it fails to adequately protect against concurrent access or manipulation between the validation check and the subsequent file operation. This creates an exploitable window where an attacker can:

1. Replace a legitimate file with a symbolic link pointing to a sensitive target file
2. Modify file ownership during the race window
3. Delete or overwrite files that should be protected

The lack of atomic file operations or proper locking mechanisms enables these attack scenarios in multi-process environments typical of HPC clusters.

Attack Vector

The attack vector is local, requiring the attacker to have existing low-privilege access to a system running a vulnerable Slurm installation. The exploitation involves:

1. Identifying predictable file operations performed by Slurm daemons or utilities
2. Creating a race condition by rapidly manipulating files (e.g., replacing regular files with symbolic links) during the window between check and use
3. Winning the race to redirect file operations to attacker-controlled targets

Due to the high complexity required to reliably exploit race conditions, successful exploitation depends on timing and system load conditions. However, in busy HPC environments with frequent job submissions, opportunities for exploitation increase.

The vulnerability allows an attacker to achieve high impact on confidentiality, integrity, and availability—reading sensitive files by redirecting reads, corrupting or overwriting critical files, or deleting essential system components.

Detection Methods for CVE-2023-41914

Indicators of Compromise

• Unexpected symbolic links appearing in Slurm working directories or spool areas
• Unusual file permission or ownership changes on Slurm-related files
• Log entries showing file operation failures or permission denied errors for normally accessible files
• Evidence of rapid file creation/deletion cycles in monitored Slurm directories

Detection Strategies

• Monitor Slurm spool directories (/var/spool/slurm/) for unexpected symbolic link creation
• Implement file integrity monitoring (FIM) on critical Slurm configuration and state files
• Audit system calls related to file operations from Slurm processes using tools like auditd
• Deploy behavioral analysis to detect anomalous file access patterns from Slurm daemons

Monitoring Recommendations

• Enable verbose logging in Slurm configuration to capture file operation details
• Configure system auditing to track symlink(), unlink(), and chown() system calls in Slurm directories
• Establish baseline behavior for Slurm file operations and alert on deviations
• Review Slurm controller and compute node logs regularly for unusual activity patterns

How to Mitigate CVE-2023-41914

Immediate Actions Required

• Upgrade SchedMD Slurm to version 23.02.6 or later for 23.02.x branch installations
• Upgrade SchedMD Slurm to version 22.05.10 or later for 22.05.x branch installations
• Review and restrict filesystem permissions on Slurm spool and configuration directories
• Audit user accounts with access to Slurm systems for unnecessary privileges

Patch Information

SchedMD has released security patches addressing this vulnerability. Administrators should upgrade to the following fixed versions:

• Slurm 23.02.x branch: Upgrade to 23.02.6 or later
• Slurm 22.05.x branch: Upgrade to 22.05.10 or later

Detailed patch information and release notes are available from the SLURM Announcement Mailing List. Fedora users should apply package updates as announced in the Fedora Package Announcement.

For additional security information, consult the SchedMD Security Information page.

Workarounds

• Restrict access to Slurm spool directories using strict filesystem permissions
• Mount Slurm working directories with nosymfollow option where supported to prevent symbolic link attacks
• Implement mandatory access control (MAC) policies using SELinux or AppArmor to limit Slurm daemon file operations
• Consider isolating Slurm infrastructure from general user access until patches can be applied

# Example: Restrict Slurm spool directory permissions
chmod 750 /var/spool/slurm
chown slurm:slurm /var/spool/slurm

# Example: Enable audit logging for Slurm directories
auditctl -w /var/spool/slurm -p wa -k slurm_file_monitor