CVE-2026-27878 Overview
CVE-2026-27878 is a resource exhaustion vulnerability ([CWE-400]) in Grafana Tempo, the open-source distributed tracing backend. The flaw resides in how the Tempo query engine processes TraceQL queries that include an exemplars hint. An authenticated user can submit a query with an excessively large exemplars hint value, causing the Tempo instance to allocate an unbounded amount of memory. The resulting out-of-memory (OOM) condition crashes the service and produces a denial of service (DoS) against tracing pipelines that depend on it.
Critical Impact
An authenticated attacker can crash Grafana Tempo instances over the network through a single crafted TraceQL query, disrupting observability and trace ingestion across dependent services.
Affected Products
- Grafana Tempo (distributed tracing backend)
- Deployments exposing the TraceQL query API to authenticated users
- Observability stacks integrating Tempo with Grafana for trace visualization
Discovery Timeline
- 2026-06-19 - CVE-2026-27878 published to the National Vulnerability Database
- 2026-06-23 - Last updated in NVD database
Technical Details for CVE-2026-27878
Vulnerability Analysis
The vulnerability arises in the TraceQL query path of Grafana Tempo. TraceQL supports an exemplars hint that instructs the query engine on how many exemplar traces to return alongside metric-style results. The query engine does not bound or validate the exemplars hint value before sizing internal data structures. When a client submits a query with a very large hint, Tempo allocates memory proportional to that value, regardless of the actual number of exemplars that would be produced.
The condition is reachable over the network and only requires low privileges, since any authenticated user with TraceQL query access can supply the hint. There is no impact on confidentiality or integrity, but availability impact is high because the affected process terminates with an OOM error. In replicated deployments, repeated queries can crash multiple query frontends or queriers in succession.
Root Cause
The root cause is missing input validation on a numeric query parameter. The exemplars hint is treated as a sizing hint for allocation rather than as untrusted input that must be capped and sanitized. Without an upper bound enforced before allocation, the query engine exposes a direct path from a user-controlled integer to large heap allocations, which is the pattern described by [CWE-400] Uncontrolled Resource Consumption.
Attack Vector
Exploitation requires network access to the Tempo query API and a valid set of credentials. The attacker issues a TraceQL query against a tenant they can already query, embedding an oversized exemplars hint. The Tempo process attempts the allocation, exhausts available memory, and is terminated by the operating system or the container runtime. Refer to the Grafana Security Advisory CVE-2026-27878 for vendor-specific details on affected versions and fixed releases.
Detection Methods for CVE-2026-27878
Indicators of Compromise
- Tempo querier or query-frontend processes terminated with OOMKilled status in Kubernetes or out-of-memory signals in systemd logs.
- Sudden spikes in resident set size (RSS) on Tempo pods correlated with inbound TraceQL queries.
- TraceQL query logs containing unusually large numeric values in the exemplars hint field.
- Repeated restarts of Tempo components without corresponding deployment changes.
Detection Strategies
- Parse Tempo query access logs for TraceQL statements containing exemplars hints and flag values above an organization-defined threshold.
- Correlate Tempo process restarts with the identity of the authenticated user who submitted the preceding query.
- Alert on memory utilization on Tempo pods crossing a high-water mark within a short observation window.
Monitoring Recommendations
- Export Tempo container memory metrics and OOM events to a centralized log and metrics platform for retention and correlation.
- Monitor authentication and authorization logs for Tempo to attribute abusive queries to specific user or service accounts.
- Track query latency and error rates per tenant to detect tenants generating expensive queries.
How to Mitigate CVE-2026-27878
Immediate Actions Required
- Upgrade Grafana Tempo to a fixed release as identified in the Grafana Security Advisory CVE-2026-27878.
- Restrict TraceQL query access to trusted users and remove unused or stale credentials from tenants exposed to the query API.
- Enforce per-tenant query limits and memory limits on Tempo querier and query-frontend deployments to contain abusive queries.
Patch Information
Grafana Labs has published advisory details and fixed versions for Tempo. Consult the Grafana Security Advisory CVE-2026-27878 for the list of patched releases and apply the upgrade through your standard deployment workflow, whether Helm chart, Tanka, or container image tag update.
Workarounds
- Place an authenticating proxy in front of the Tempo query API and reject TraceQL queries whose exemplars hint exceeds a safe threshold.
- Configure Kubernetes resource limits and OOMScoreAdj settings so that crashed Tempo pods restart quickly and do not destabilize other workloads on the node.
- Reduce the blast radius by running multiple replicas of the query-frontend and querier behind a load balancer with health checks.
- Audit and rotate credentials for any account that does not require TraceQL query access.
Disclaimer: This content was generated using AI. While we strive for accuracy, please verify critical information with official sources.
