CVE-2024-3574: Scrapy Information Disclosure Vulnerability

CVE-2024-3574 Overview

CVE-2024-3574 is an information disclosure vulnerability in Scrapy, an open source web crawling and scraping framework written in Python. In version 2.10.1 and earlier, the framework fails to strip the Authorization header when following HTTP redirects that cross domain boundaries. Credentials intended for the original server are transmitted to the redirect target, exposing them to unauthorized third-party hosts. An attacker who controls or compromises a redirect destination can capture these credentials and potentially hijack the associated account. The issue is categorized under [CWE-200: Exposure of Sensitive Information to an Unauthorized Actor].

Critical Impact
Authorization credentials configured in Scrapy spiders can be silently forwarded to attacker-controlled domains during cross-origin redirects, enabling account takeover against the originally targeted service.

Affected Products

Scrapy versions up to and including 2.10.1
Any Python application or crawler embedding vulnerable Scrapy releases
Downstream scraping pipelines depending on Scrapy's default RedirectMiddleware

Discovery Timeline

2024-04-16 - CVE-2024-3574 published to NVD
2026-06-17 - Last updated in NVD database

Technical Details for CVE-2024-3574

Vulnerability Analysis

Scrapy's RedirectMiddleware rebuilds a follow-up request when a server responds with a 3xx status. Prior to the patch, the middleware only inspected the Cookie header before deciding whether to strip credentials on cross-domain hops. The Authorization header, which carries HTTP Basic, Bearer, or custom tokens, was preserved unconditionally and re-sent to whatever host the Location header indicated. An attacker who owns a domain that the scraped site redirects to, or who can inject a redirect through an open redirect flaw upstream, receives the bearer token or basic credentials verbatim. The disclosure violates the cors-non-wildcard-request-header-name guidance in the Fetch specification, which explicitly requires the Authorization header to be treated as sensitive across origins.

Root Cause

The defect lies in scrapy/downloadermiddlewares/redirect.py. The logic that scrubbed sensitive headers on cross-netloc redirects only checked for Cookie and had no equivalent branch for Authorization. Because Scrapy applies Authorization values from spider settings or Request.headers on every dispatched request, redirects inherited the header regardless of destination.

Attack Vector

Exploitation requires no authentication or user interaction. An attacker configures a domain to respond with an HTTP 30x redirect pointing to a host they control, then induces the Scrapy target to fetch a URL under their influence. Any spider that adds an Authorization header for the intended service leaks the credential on the follow-up request. The attacker can then replay the token against the legitimate API to impersonate the account.

python

         **kwargs,
         cookies=None,
     )
-    if "Cookie" in redirect_request.headers:
+    has_cookie_header = "Cookie" in redirect_request.headers
+    has_authorization_header = "Authorization" in redirect_request.headers
+    if has_cookie_header or has_authorization_header:
         source_request_netloc = urlparse_cached(source_request).netloc
         redirect_request_netloc = urlparse_cached(redirect_request).netloc
         if source_request_netloc != redirect_request_netloc:
-            del redirect_request.headers["Cookie"]
+            if has_cookie_header:
+                del redirect_request.headers["Cookie"]
+            # https://fetch.spec.whatwg.org/#ref-for-cors-non-wildcard-request-header-name
+            if has_authorization_header:
+                del redirect_request.headers["Authorization"]
     return redirect_request

Source: Scrapy patch commit 5bcb8fd

Detection Methods for CVE-2024-3574

Indicators of Compromise

Outbound HTTPS requests from Scrapy hosts to unexpected domains carrying Authorization: Basic or Authorization: Bearer headers.
Access log entries at third-party services showing valid API tokens originating from crawler IP ranges.
Authentication events on protected APIs sourced from crawler infrastructure but with unusual Referer or Origin values.

Detection Strategies

Inventory Python environments and flag any installation of scrapy at version <= 2.10.1 using pip list or SBOM scans.
Inspect proxy or egress logs for HTTP 30x chains where the destination netloc differs from the initial request but the Authorization header is still present.
Enable Scrapy's SPIDER_MIDDLEWARES debug logging and audit redirect chains that terminate outside the intended target domain.

Monitoring Recommendations

Alert on Scrapy job runs that follow redirects into domains not on an allowlist tied to the crawl target.
Rotate and monitor any API tokens that have been used by Scrapy spiders since the vulnerable version was deployed.
Correlate crawler egress traffic with DNS logs to detect suspicious redirect targets registered on low-reputation infrastructure.

How to Mitigate CVE-2024-3574

Immediate Actions Required

Upgrade Scrapy to version 2.11.1 or later, which drops the Authorization header on cross-domain redirects.
Rotate every credential and API token that has been supplied to Scrapy spiders via Authorization headers.
Audit historical crawl logs to identify redirect chains that may have exfiltrated credentials to unintended hosts.

Patch Information

The fix ships in Scrapy 2.11.1 via commit 5bcb8fd. The updated RedirectMiddleware in scrapy/downloadermiddlewares/redirect.py now strips both Cookie and Authorization when the source and redirect netloc values differ. Full details are documented in the Huntr bounty report and the upstream GHSA-cw9j-q3vf-hrrv advisory.

Workarounds

Disable Scrapy's default redirect handling by setting REDIRECT_ENABLED = False in settings.py until the upgrade is applied.
Implement a custom downloader middleware that removes the Authorization header whenever response.status is in the 300 range and Location points to a different domain.
Restrict spiders to a per-run allowlist of domains using OffsiteMiddleware and allowed_domains so that redirects to attacker-controlled hosts are dropped before dispatch.

bash

# Upgrade Scrapy to the patched release
pip install --upgrade 'scrapy>=2.11.1'

# Verify the installed version
python -c "import scrapy; print(scrapy.__version__)"

# Temporary hardening in settings.py
# REDIRECT_ENABLED = False