CVE-2024-1892: Scrapy XMLFeedSpider ReDoS Vulnerability

CVE-2024-1892 Overview

CVE-2024-1892 is a Regular Expression Denial of Service (ReDoS) vulnerability in the XMLFeedSpider class of the Scrapy web crawling framework. The flaw resides in the XML parsing routines used by scrapy.utils.iterators.xmliter, where inefficient regular expression patterns exhibit catastrophic backtracking against crafted input. An attacker who can influence the XML content processed by a Scrapy spider can trigger excessive CPU and memory consumption, causing the crawler process to hang. This makes any service that relies on Scrapy for XML feed processing unresponsive. The vulnerability is tracked under [CWE-1333: Inefficient Regular Expression Complexity].

Critical Impact
A single malicious XML response can exhaust CPU resources on a Scrapy worker, halting crawl operations and denying service to downstream consumers of the scraped data.

Affected Products

Scrapy versions prior to 2.11.1 using XMLFeedSpider
Applications importing scrapy.utils.iterators.xmliter
Any Scrapy pipeline consuming untrusted XML feeds

Discovery Timeline

2024-02-28 - CVE-2024-1892 published to NVD
2026-06-17 - Last updated in NVD database

Technical Details for CVE-2024-1892

Vulnerability Analysis

The vulnerability affects Scrapy's XMLFeedSpider, a spider class designed to iterate over XML feeds node by node. The underlying iterator, scrapy.utils.iterators.xmliter, uses hand-written regular expressions to identify node boundaries within XML documents. Under specific input structures, these expressions exhibit exponential-time backtracking behavior. An attacker who controls the response body served to a Scrapy crawler can supply crafted XML that forces the regex engine into a pathological matching state. The crawler process consumes CPU indefinitely and blocks further request processing.

Because Scrapy is commonly deployed as long-running background workers ingesting third-party feeds, a single hostile endpoint can stall an entire crawl pipeline. The impact is confined to availability: no code execution, information disclosure, or data tampering occurs.

Root Cause

The root cause is inefficient regular expression complexity in xmliter. The regex used to locate XML nodes contains ambiguous alternations and nested quantifiers that produce catastrophic backtracking when the input contains many overlapping partial matches. A secondary regex flaw with the same class of issue was identified in scrapy.utils.response.open_in_browser.

Attack Vector

Exploitation requires the victim spider to fetch attacker-influenced XML. This occurs when a Scrapy job crawls user-submitted URLs, third-party feeds, or endpoints that an adversary can compromise or spoof. User interaction is required in the sense that the crawl must target the malicious resource. Once the response body reaches xmliter, parsing stalls and consumes memory until the process is terminated.

text

# Patch excerpt from docs/news.rst
Security bug fixes
~~~~~~~~~~~~~~~~~~

-   Addressed ReDoS vulnerabilities:

    -   scrapy.utils.iterators.xmliter is now deprecated in favor of
        scrapy.utils.iterators.xmliter_lxml, which
        XMLFeedSpider now uses.

    -   Fixed regular expressions in the implementation of the
        scrapy.utils.response.open_in_browser function.

Source: Scrapy commit 479619b

Detection Methods for CVE-2024-1892

Indicators of Compromise

Scrapy worker processes stuck at 100% CPU utilization while processing a single response
Growing resident memory of Python processes running XMLFeedSpider jobs
Crawl jobs that stop emitting items or logs shortly after fetching an XML feed
Scrapy telemetry showing response download completion followed by prolonged parsing time

Detection Strategies

Inventory Python dependencies for scrapy versions earlier than 2.11.1 using SBOM tooling or pip list
Alert on Scrapy processes whose CPU time exceeds a baseline for XML feed jobs
Instrument spiders with per-response timing to flag parsing operations that exceed a configured threshold
Review spider source for direct imports of scrapy.utils.iterators.xmliter, which is now deprecated

Monitoring Recommendations

Forward Scrapy worker resource metrics (CPU, RSS, event loop latency) to a centralized observability platform
Enable request and response logging with timestamps to correlate hangs with specific source URLs
Track outbound domains crawled by XMLFeedSpider jobs and treat sudden availability drops as suspicious
Set process-level watchdog timers that terminate spiders exceeding an expected parse duration

How to Mitigate CVE-2024-1892

Immediate Actions Required

Upgrade Scrapy to version 2.11.1 or later where XMLFeedSpider uses xmliter_lxml
Replace direct calls to scrapy.utils.iterators.xmliter with scrapy.utils.iterators.xmliter_lxml
Constrain crawl scope so XMLFeedSpider only fetches trusted feeds until the upgrade is complete
Apply the DOWNLOAD_MAXSIZE and DOWNLOAD_TIMEOUT settings to bound response size and processing time

Patch Information

The fix is delivered in commit 479619b, which deprecates xmliter in favor of the lxml-based xmliter_lxml and repairs the regular expressions used by open_in_browser. Full context is available in the Huntr bounty report and the upstream GHSA-cc65-xxvf-f7r9 advisory referenced in the release notes.

Workarounds

Switch spiders to use xmliter_lxml directly instead of xmliter if an immediate framework upgrade is not possible
Wrap XML parsing calls in a subprocess or thread with a hard timeout to bound worst-case CPU consumption
Validate or size-limit response bodies before passing them to XMLFeedSpider parsers
Restrict allowed_domains on affected spiders to sources under organizational control

bash

# Upgrade Scrapy to a patched release
pip install --upgrade 'scrapy>=2.11.1'

# Verify the installed version
python -c "import scrapy; print(scrapy.__version__)"

# Bound response size and download time in settings.py
# DOWNLOAD_MAXSIZE = 10_485_760   # 10 MB
# DOWNLOAD_TIMEOUT = 30           # seconds