CVE-2026-105782: Scrapy Python Framework RCE Vulnerability

CVE-2026-105782 Overview

CVE-2026-105782 is an unsafe reflection vulnerability [CWE-470] in Scrapy, a widely used Python web crawling and scraping framework. The flaw resides in RefererMiddleware within scrapy/spidermiddlewares/referer.py and affects versions from 1.4.0 through 2.14.1. The middleware treated any Referrer-Policy response header value that resembled a Python import path as a referrer policy class, imported the referenced object, and then invoked it. A malicious website can supply a callable such as sys.exit and terminate the crawler mid-response. The issue is fixed in Scrapy 2.14.2.

Critical Impact

Any website crawled by a vulnerable Scrapy spider can forcibly terminate the crawler process by returning a crafted Referrer-Policy header, breaking data collection pipelines.

Affected Products

• Scrapy versions 1.4.0 through 2.14.1
• Python-based crawlers and scraping pipelines depending on Scrapy
• Automated data collection and OSINT workflows built on vulnerable Scrapy releases

Discovery Timeline

• 2026-10-06 - CVE-2026-105782 published to NVD
• 2026-10-06 - Last updated in NVD database
• Scrapy 2.14.2 - Fix released via commits 945b787a and b6e5c58a, documented in GHSA-cwxj-rr6w-m6w7

Technical Details for CVE-2026-105782

Vulnerability Analysis

Scrapy's RefererMiddleware is responsible for honoring the Referrer-Policy header defined by the W3C Referrer Policy specification. In vulnerable releases, the middleware accepted standard policy tokens such as no-referrer or strict-origin-when-cross-origin, but it also accepted strings shaped like Python import paths. When such a string was received, Scrapy resolved it with its import utilities, loaded the referenced object, and called it as if it were a ReferrerPolicy subclass. This conflation of policy names with arbitrary importable callables turns an untrusted HTTP header into a trigger for in-process function execution. The practical outcome documented in GHSA-cwxj-rr6w-m6w7 is denial of service: a response containing Referrer-Policy: sys.exit causes the crawler process to terminate.

Root Cause

The root cause is unsafe reflection [CWE-470]. The middleware used attacker-controlled input to select which object to import and call, without validating that the string belonged to an allow-list of referrer policy classes. Any importable object reachable in the Scrapy process became a candidate target.

Attack Vector

Exploitation requires no authentication and no user interaction beyond the normal act of crawling. An attacker only needs to control a web resource that the Scrapy spider fetches. When the spider processes the response, the malicious Referrer-Policy header is parsed and the referenced callable is invoked inside the crawler process.

# Patch excerpt from Scrapy 2.14.2 release notes (docs/news.rst)
# Source: https://github.com/scrapy/scrapy/commit/b6e5c58ae707a3d4bb491537b5519534050047e0

Scrapy 2.14.2 (unreleased)
--------------------------

Security bug fixes
~~~~~~~~~~~~~~~~~~

-   Values from the ``Referrer-Policy`` header of HTTP responses are no longer
    executed as Python callables. See the `cwxj-rr6w-m6w7`_ security advisory
    for details.

New features
~~~~~~~~~~~~

-   Added a new setting, REFERER_POLICIES, to allow customizing
    supported referrer policies.

Detection Methods for CVE-2026-105782

Indicators of Compromise

• Scrapy crawler processes exiting unexpectedly immediately after fetching an external URL, with no traceback beyond a SystemExit or similar interpreter shutdown.
• HTTP responses in crawl logs containing non-standard Referrer-Policy header values that resemble Python dotted import paths, for example sys.exit, os._exit, or builtins.quit.
• Spiders that previously ran to completion now terminate mid-run after visiting newly introduced domains or redirects.

Detection Strategies

• Inventory running Scrapy versions across build pipelines and scraping hosts; flag anything from 1.4.0 up to and including 2.14.1 as vulnerable.
• Enable response header logging in Scrapy middlewares and alert on Referrer-Policy values that are not in the W3C-defined token list.
• Review proxy or egress logs for responses whose Referrer-Policy header contains a period (.), which is characteristic of a Python import path rather than a valid policy token.

Monitoring Recommendations

• Track crawler process lifetimes and exit codes; abrupt SystemExit terminations should raise alerts.
• Correlate spider crashes with the last-fetched URL and response headers to attribute terminations to malicious responses.
• Monitor dependency manifests (requirements.txt, pyproject.toml, lockfiles) in source control for pinned Scrapy versions below 2.14.2.

How to Mitigate CVE-2026-105782

Immediate Actions Required

• Upgrade Scrapy to version 2.14.2 or later across all crawler hosts, container images, and CI/CD build artifacts.
• Audit custom spider middlewares that subclass or wrap RefererMiddleware to ensure they do not reintroduce the unsafe import behavior.
• Rotate any long-lived crawl jobs through the patched release before re-enabling scheduled runs against untrusted targets.

Patch Information

The fix is delivered in Scrapy 2.14.2 via commits 945b787a and b6e5c58a. Values from the Referrer-Policy response header are no longer resolved as Python callables. A new REFERRER_POLICIES setting lets operators map approved policy names to scrapy.spidermiddlewares.referer.ReferrerPolicy subclasses, or disable a given policy name by mapping it to None.

Workarounds

• If immediate upgrade is not possible, disable scrapy.spidermiddlewares.referer.RefererMiddleware in the spider's SPIDER_MIDDLEWARES setting and set the referrer policy statically via REFERRER_POLICY.
• Deploy a custom downloader middleware that strips or normalizes the Referrer-Policy response header to the W3C token set before RefererMiddleware processes it.
• Run crawlers under a process supervisor configured to restart terminated spiders, limiting denial-of-service impact until the patch is applied.

# Upgrade Scrapy to the patched release
pip install --upgrade "scrapy>=2.14.2"

# Verify installed version
python -c "import scrapy; print(scrapy.__version__)"

# Optional: pin the fix in requirements.txt
echo 'scrapy>=2.14.2' >> requirements.txt