CVE-2026-55520: Protego Robots.txt Parser DoS Vulnerability

CVE-2026-55520 Overview

Protego is a pure-Python robots.txt parser with support for modern conventions used by web crawlers, notably the Scrapy framework. Versions prior to 0.6.2 contain a Regular Expression Denial of Service (ReDoS) vulnerability in the URL pattern matching logic. The _URLPattern._prepare_pattern_for_regex method translates every asterisk in an Allow or Disallow directive into a lazy regular-expression wildcard. A directive containing many asterisks creates exponential backtracking during regex evaluation. An attacker who controls a robots.txt file served to the crawler can trigger this behavior. The issue is fixed in version 0.6.2.

Critical Impact
A malicious server can serve a crafted robots.txt file that forces protego.Protego.can_fetch to consume attacker-controlled CPU time, denying service to the crawler process.

Affected Products

Protego versions prior to 0.6.2
Applications using Protego for robots.txt parsing (including Scrapy-based crawlers)
Vulnerable code path: src/protego/_urlpattern.py

Discovery Timeline

2026-08-28 - CVE-2026-55520 published to NVD
2026-08-31 - Last updated in NVD database

Technical Details for CVE-2026-55520

Vulnerability Analysis

The vulnerability is a ReDoS flaw classified under [CWE-400] Uncontrolled Resource Consumption. Protego compiles URL patterns from robots.txtAllow and Disallow directives into Python regular expressions. Each * wildcard is translated into a lazy quantifier in the compiled pattern. When multiple lazy wildcards appear in a single directive, matching a near-miss URL forces the regex engine into catastrophic backtracking. The runtime grows exponentially with the number of wildcards.

The attack requires no authentication. A crawler that fetches a hostile server's robots.txt file passes the content to protego.Protego.parse, then invokes can_fetch against candidate URLs. The can_fetch call blocks on regex evaluation, stalling the crawler.

Root Cause

The root cause lives in src/protego/_urlpattern.py, where _URLPattern._prepare_pattern_for_regex mapped each * character to a lazy wildcard expression without bounding the total number of wildcards or the resulting regex complexity. Python's re engine uses backtracking, so multiple adjacent lazy wildcards followed by a partial match produce exponential state exploration.

Attack Vector

Exploitation requires an attacker to control the robots.txt content served to a Protego-based crawler. The attacker publishes a crafted file with directives containing many * characters, then induces the crawler to visit URLs that partially match those patterns. The following patch excerpt from the upstream project documents the fix:

text

0.6.2 (unreleased)
==================

-   Fixed a ReDoS (regular expression denial of service) vulnerability: URL
    patterns from ``robots.txt`` ``Allow`` and ``Disallow`` directives were
    compiled into regular expressions, where multiple ``*`` wildcards could
    cause exponential backtracking. A server could exploit this to cause denial
    of service by serving a crafted ``robots.txt`` file. Wildcard matching is
    now performed without regular expressions.

Source: GitHub Commit 7859401

Detection Methods for CVE-2026-55520

Indicators of Compromise

Crawler processes with sustained high CPU usage while parsing or matching robots.txt content
robots.txt files containing Allow or Disallow directives with numerous consecutive * characters
Stalled or timed-out Scrapy jobs immediately after fetching a target's robots.txt
Python stack traces showing extended time inside re module calls originating from protego._urlpattern

Detection Strategies

Inventory all Python environments and identify installations of protego below version 0.6.2 using pip list or SBOM tooling
Add pre-parse validation that rejects or truncates robots.txt directives containing more than a small number of wildcard characters
Instrument crawler workers with per-request CPU and wall-clock timeouts to surface anomalous matching times

Monitoring Recommendations

Monitor crawler telemetry for outlier can_fetch execution times per host
Log the length and wildcard count of Allow and Disallow directives during robots.txt ingestion
Alert on repeated crawler worker restarts or task timeouts tied to a specific target domain

How to Mitigate CVE-2026-55520

Immediate Actions Required

Upgrade protego to version 0.6.2 or later across all crawler and scraping environments
Audit dependency manifests (requirements.txt, pyproject.toml, Pipfile.lock) for pinned versions below 0.6.2
Rebuild and redeploy container images that bundle vulnerable Protego versions

Patch Information

The fix is available in Protego 0.6.2. The upstream commit removes the regex-based wildcard implementation and performs wildcard matching without regular expressions, eliminating the backtracking primitive. Full details are in the GitHub Security Advisory GHSA-wjmf-p669-5m5p and the Protego 0.6.2 Release Notes.

Workarounds

Apply a wrapper around Protego.parse that rejects robots.txt files whose directives contain excessive wildcard characters
Enforce a strict wall-clock timeout on can_fetch calls using signal-based or subprocess isolation
Restrict crawler targets to a trusted allowlist until the upgrade is deployed

bash

# Upgrade Protego to the fixed version
pip install --upgrade 'protego>=0.6.2'

# Verify the installed version
python -c "import protego; print(protego.__version__)"