CVE-2024-36129: Opentelemetry Configgrpc DOS Vulnerability

CVE-2024-36129 Overview

CVE-2024-36129 is an unsafe decompression vulnerability affecting the OpenTelemetry Collector, a vendor-agnostic telemetry data processing framework. This vulnerability allows unauthenticated remote attackers to cause a denial of service condition by triggering excessive memory consumption through crafted compressed payloads. The OpenTelemetry Collector is widely deployed in cloud-native observability stacks, making this vulnerability particularly concerning for organizations relying on distributed tracing and metrics collection.

Critical Impact

Unauthenticated attackers can remotely crash OpenTelemetry Collector instances through memory exhaustion, potentially disrupting observability infrastructure and creating blind spots in monitoring capabilities.

Affected Products

• OpenTelemetry Collector versions prior to 0.102.1
• OpenTelemetry confighttp module versions prior to 0.102.0
• OpenTelemetry configgrpc module versions prior to 0.102.1

Discovery Timeline

• 2024-06-05 - CVE-2024-36129 published to NVD
• 2024-11-21 - Last updated in NVD database

Technical Details for CVE-2024-36129

Vulnerability Analysis

This vulnerability is classified under CWE-119 (Improper Restriction of Operations within the Bounds of a Memory Buffer) and stems from insufficient validation during the decompression of incoming telemetry data. The OpenTelemetry Collector accepts compressed payloads via HTTP and gRPC endpoints to optimize network bandwidth, but the decompression routines fail to properly constrain the expanded data size. An attacker can exploit this by sending specially crafted compressed data that expands to consume excessive memory when decompressed, a technique commonly known as a "decompression bomb" or "zip bomb" attack.

The vulnerability is accessible over the network without authentication, meaning any attacker with network access to an exposed collector endpoint can trigger the condition. The attack requires no user interaction and can be executed with low complexity, though the impact is limited to availability—confidentiality and integrity remain unaffected.

Root Cause

The root cause lies in the confighttp and configgrpc modules' handling of compressed HTTP and gRPC requests. These modules support compression algorithms like gzip and zstd for efficient data transfer but lacked proper bounds checking on the decompressed output size. Without limits on the decompression ratio or maximum uncompressed size, a small compressed payload could expand into gigabytes of data in memory, exhausting available resources.

Attack Vector

An attacker exploits this vulnerability by sending a malicious compressed payload to an exposed OpenTelemetry Collector endpoint. The attack flow involves:

1. Identifying an accessible OpenTelemetry Collector HTTP or gRPC receiver endpoint
2. Crafting a highly compressed payload with an extreme compression ratio (e.g., a small payload that decompresses to gigabytes)
3. Sending the malicious payload with appropriate Content-Encoding headers (gzip, zstd, etc.)
4. The collector attempts to decompress the payload, consuming excessive memory
5. Memory exhaustion causes the collector process to crash or become unresponsive

The attack does not require authentication, as telemetry receivers are typically designed to accept data from various instrumented services. In cloud-native environments where collectors are exposed or accessible from pods, this creates significant attack surface.

Detection Methods for CVE-2024-36129

Indicators of Compromise

• Sudden spikes in memory usage on systems running OpenTelemetry Collector
• Collector process crashes with out-of-memory (OOM) errors in logs
• Unusual incoming requests with compressed payloads to collector endpoints (ports 4317 for gRPC, 4318 for HTTP)
• Repeated collector restarts observed in container orchestration logs

Detection Strategies

• Monitor OpenTelemetry Collector memory consumption for anomalous growth patterns using infrastructure monitoring tools
• Implement logging to capture and analyze incoming request sizes and compression headers
• Configure alerting on collector process restarts or OOM killer events in container environments
• Review network traffic logs for unusually small requests with compression headers targeting collector endpoints

Monitoring Recommendations

• Set up memory utilization thresholds and alerts for OpenTelemetry Collector deployments
• Enable detailed access logging on HTTP/gRPC receiver endpoints to track request metadata
• Implement network-level monitoring to detect potential decompression bomb attack patterns
• Use container orchestration health checks to detect and alert on repeated pod restarts

How to Mitigate CVE-2024-36129

Immediate Actions Required

• Upgrade OpenTelemetry Collector to version 0.102.1 or later immediately
• If using the confighttp module independently, upgrade to version 0.102.0 or later
• If using the configgrpc module independently, upgrade to version 0.102.1 or later
• Restrict network access to collector endpoints using firewall rules or network policies

Patch Information

OpenTelemetry has released fixes in the following versions:

• OpenTelemetry Collector: Version 0.102.1 addresses this vulnerability
• confighttp module: Version 0.102.0 includes the fix
• configgrpc module: Version 0.102.1 includes the fix

The patches implement proper bounds checking and limits on decompression operations. For detailed information, refer to the GitHub Security Advisory GHSA-c74f-6mfw-mm4v, GitHub Pull Request #10289, and the OpenTelemetry Blog on CVE-2024-36129.

Workarounds

• Place a reverse proxy or API gateway in front of collector endpoints that can limit request body sizes
• Implement network segmentation to restrict which services can send data to the collector
• Configure resource limits (memory) on collector containers to prevent host-level impact, though this will still result in collector crashes
• Disable unused compression algorithms if your deployment allows uncompressed telemetry data

# Example: Configure Kubernetes resource limits for OpenTelemetry Collector
# This limits blast radius but does not prevent the DoS
kubectl patch deployment otel-collector -p '
spec:
  template:
    spec:
      containers:
      - name: otel-collector
        resources:
          limits:
            memory: "2Gi"
          requests:
            memory: "512Mi"
'