CVE-2024-3572: Scrapy XXE Vulnerability

CVE-2024-3572 Overview

CVE-2024-3572 affects the scrapy/scrapy web crawling framework. The vulnerability stems from the use of lxml.etree.fromstring to parse untrusted XML data without disabling external entity resolution. Attackers who supply crafted XML responses to a Scrapy spider can trigger XML External Entity (XXE) processing. This enables denial of service, local file access, outbound network connections from the crawler host, and firewall traversal. The issue is tracked under [CWE-409] and carries a network attack vector with no authentication or user interaction required. Scrapy is widely used for large-scale web scraping and data extraction, making crawler hosts an attractive target for XXE-driven reconnaissance.

Critical Impact
A malicious XML response can force a Scrapy crawler to disclose local files, exhaust memory, or initiate arbitrary outbound connections from the crawler host.

Affected Products

Scrapy framework (scrapy/scrapy) versions prior to the patched release
Python applications embedding Scrapy for XML feed or sitemap ingestion
Automated crawlers processing untrusted XML responses via lxml

Discovery Timeline

2024-04-16 - CVE-2024-3572 published to the National Vulnerability Database
2026-06-17 - Last updated in NVD database

Technical Details for CVE-2024-3572

Vulnerability Analysis

Scrapy invokes lxml.etree.fromstring on XML payloads returned by remote servers without configuring a hardened parser. The default lxml parser resolves external entity declarations, external DTDs, and entity expansion directives. When a crawler retrieves attacker-controlled XML, embedded <!ENTITY> declarations are dereferenced during parsing. This behavior converts a passive content retrieval operation into an active file read or network request executed with the privileges of the crawler process. The attack requires only that the target crawler process an XML document sourced from a host the attacker controls or can influence.

Root Cause

The root cause is missing parser hardening. lxml.etree.fromstring accepts an optional parser argument that can disable resolve_entities, load_dtd, and no_network. Scrapy did not construct such a parser before deserializing untrusted XML. As a result, entity references such as SYSTEM "file:///etc/passwd" or SYSTEM "http://internal-service/" are resolved during tree construction.

Attack Vector

An attacker hosts a malicious XML document at a URL a Scrapy spider will fetch. This can be a sitemap, an RSS feed, or any XML response the spider parses. When Scrapy parses the response, external entities defined in the payload trigger local file reads, outbound HTTP requests to internal services, or resource exhaustion via billion-laughs style expansion. The attack requires no authentication and no user interaction beyond the crawler visiting the malicious URL.

text

// Security patch excerpt from docs/news.rst
 Security bug fixes
 ~~~~~~~~~~~~~~~~~~

+-   :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
+    to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
+    advisory`_ for more information.
+
+-   Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, the
+    deprecated ``scrapy.downloadermiddlewares.decompression`` module has been
+    removed.

Source: Scrapy Security Commit 809bfac. The referenced merge includes multiple hardening changes shipped alongside the XXE fix.

Detection Methods for CVE-2024-3572

Indicators of Compromise

Outbound HTTP or DNS requests from crawler hosts to internal-only addresses shortly after fetching an external XML resource
Crawler process reads of sensitive local files such as /etc/passwd, /proc/self/environ, or Windows configuration files
XML responses containing <!DOCTYPE, <!ENTITY, or SYSTEM declarations logged by Scrapy middlewares
Sudden memory or CPU spikes in the crawler process while parsing a single XML response

Detection Strategies

Inspect Scrapy access logs for XML responses that contain DTD or entity declarations and correlate with the fetching spider
Alert on crawler hosts initiating connections to RFC1918 addresses or metadata endpoints such as 169.254.169.254
Baseline normal file access patterns for the Scrapy service account and flag reads of system credential files

Monitoring Recommendations

Forward Scrapy application logs, host process telemetry, and network flow data into a centralized analytics platform for correlation
Track lxml version and Scrapy version across all crawler nodes and alert on unpatched instances
Monitor DNS query patterns from crawler hosts for entity-driven exfiltration channels

How to Mitigate CVE-2024-3572

Immediate Actions Required

Upgrade Scrapy to the patched release referenced in the Scrapy security commit
Audit custom spiders and middlewares for direct calls to lxml.etree.fromstring, lxml.etree.parse, or xml.etree.ElementTree on untrusted input
Restrict outbound network egress from crawler hosts to only the domains required for scraping
Run Scrapy under a low-privilege service account with no access to sensitive files

Patch Information

The upstream fix is available in the scrapy/scrapy repository. Review the Huntr Bounty Report and the GitHub commit changes for the affected code paths and validate that deployed versions include the hardened XML parser configuration.

Workarounds

Construct an lxml parser with resolve_entities=False, no_network=True, and load_dtd=False before parsing any untrusted XML in custom code
Prefer defusedxml wrappers over direct lxml calls when parsing responses from third-party sources
Apply strict content-type validation and reject XML payloads that include DTD declarations
Isolate crawler workloads on network segments without access to internal services or cloud metadata endpoints

bash

# Upgrade Scrapy to a patched release
pip install --upgrade scrapy

# Verify the installed version
python -c "import scrapy; print(scrapy.__version__)"

# Optional: install defusedxml for safer XML handling in custom parsers
pip install defusedxml