CVE-2022-21698: Prometheus Client Golang DoS Vulnerability

CVE-2022-21698 Overview

CVE-2022-21698 is a Denial of Service vulnerability affecting the Prometheus client_golang instrumentation library for Go applications. The promhttp package in client_golang provides tooling around HTTP servers and clients, and in versions prior to 1.11.1, the HTTP server is susceptible to unbounded cardinality and potential memory exhaustion when handling requests with non-standard HTTP methods.

The vulnerability allows attackers to send HTTP requests with arbitrary method names, causing the instrumentation middleware to create new metric labels for each unique method. This leads to uncontrolled growth of in-memory time series data, eventually exhausting available memory and causing service disruption.

Critical Impact

Unauthenticated remote attackers can exhaust server memory by sending requests with non-standard HTTP methods, leading to complete service denial for applications using Prometheus client_golang instrumentation.

Affected Products

• Prometheus client_golang (versions prior to 1.11.1)
• Fedora 34, 35, 36, and 37
• Fedora Extra Packages for Enterprise Linux 7.0 and 8.0
• RDO Project RDO

Discovery Timeline

• February 15, 2022 - CVE-2022-21698 published to NVD
• November 21, 2024 - Last updated in NVD database

Technical Details for CVE-2022-21698

Vulnerability Analysis

The Prometheus client_golang library provides promhttp.InstrumentHandler* middleware functions that instrument HTTP handlers with metrics. These middleware functions track various HTTP metrics, including request counts categorized by HTTP method using a method label. The vulnerability arises because the library does not validate or sanitize the HTTP method value before using it as a metric label.

When an application uses any of the promhttp.InstrumentHandler* middleware (except RequestsInFlight) and passes a metric with the method label name, each unique HTTP method value creates a new time series in memory. The HTTP/1.1 specification allows arbitrary method names, and Go's http.Request struct exposes whatever method string is provided in the request.

An attacker can exploit this by sending requests with randomized or incrementing method names, causing the Prometheus metrics registry to allocate memory for each unique method value encountered. Since there is no upper bound on the cardinality of the method label, memory consumption grows unbounded until the application crashes or becomes unresponsive.

Root Cause

The root cause is the lack of input validation on HTTP method values used as metric labels. The promhttp middleware directly uses the http.Request.Method field as a label value without checking if it matches a known, valid HTTP method (GET, POST, PUT, DELETE, etc.). This design assumes that incoming requests will use standard methods, which is not enforced by the HTTP protocol itself.

Additionally, Go's HTTP server accepts requests with any method string, passing them through to handlers without validation. Combined with the Prometheus client's unbounded label cardinality, this creates a resource exhaustion condition.

Attack Vector

The attack can be executed remotely over the network without authentication. An attacker sends HTTP requests to an instrumented endpoint using random or sequential method names instead of standard HTTP methods. Each unique method creates a new time series entry in the Prometheus metrics registry.

The attack is effective when the target application:

1. Uses any promhttp.InstrumentHandler* middleware except RequestsInFlight
2. Does not filter specific HTTP methods before the middleware
3. Passes a metric with the method label to the middleware
4. Has no firewall, load balancer, or proxy that filters non-standard HTTP methods

The attack payload consists of simple HTTP requests with custom method names, requiring minimal bandwidth to cause significant memory growth on the target server.

Detection Methods for CVE-2022-21698

Indicators of Compromise

• Unusual HTTP requests with non-standard method names in access logs (e.g., AAAA, TEST123, random strings)
• Rapidly increasing memory consumption on services using Prometheus instrumentation
• Prometheus metric cardinality alerts showing unexpected growth in time series count
• Application crashes with out-of-memory errors in Go services using client_golang

Detection Strategies

• Monitor HTTP access logs for requests with methods other than GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS, and TRACE
• Implement cardinality monitoring for Prometheus metrics, alerting on sudden increases in unique label values
• Deploy application performance monitoring to track memory usage trends and correlate with incoming request patterns
• Use network intrusion detection rules to identify HTTP requests with anomalous method values

Monitoring Recommendations

• Set up alerts for Prometheus go_memstats_alloc_bytes exceeding baseline thresholds
• Monitor the prometheus_tsdb_head_series metric for unexpected cardinality growth
• Implement log analysis to detect patterns of non-standard HTTP methods from single sources
• Configure infrastructure monitoring to alert on rapid memory consumption changes in Go services

How to Mitigate CVE-2022-21698

Immediate Actions Required

• Upgrade Prometheus client_golang to version 1.11.1 or later immediately
• Audit application code for use of promhttp.InstrumentHandler* middleware with the method label
• Configure reverse proxies or web application firewalls to allow only standard HTTP methods
• Review Prometheus metrics for signs of cardinality explosion and clean up affected time series

Patch Information

Prometheus has released version 1.11.1 of client_golang which contains a fix for this vulnerability. The patch sanitizes HTTP method values, mapping non-standard methods to a known safe value to prevent unbounded cardinality. Details of the fix can be found in GitHub Pull Request #962 and GitHub Pull Request #987. The patched release is available at GitHub Release v1.11.1.

For full security advisory details, see the GitHub Security Advisory GHSA-cg3q-j54f-5p7p.

Workarounds

• Remove the method label name from counters and gauges used with the InstrumentHandler* middleware
• Turn off affected promhttp handlers if metrics collection can be temporarily disabled
• Add custom middleware before the promhttp handler to sanitize request methods to a known set of valid values
• Use a reverse proxy or web application firewall configured to allow only standard HTTP methods (GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS)

# Example nginx configuration to restrict HTTP methods
location / {
    # Only allow standard HTTP methods
    limit_except GET POST PUT DELETE PATCH HEAD OPTIONS {
        deny all;
    }
    
    proxy_pass http://backend;
}