Prometheus High Cardinality Troubleshoot Fix: From Detection to Production Recovery

Let me tell you about the night our Prometheus cluster tried to die.

We were running a standard setup—3 nodes, ~200k active series, nothing special. Then the pager went off at 3 AM. TSDB write latency hit 5 seconds. Queries were timing out. Grafana dashboards looked like a Christmas tree of red alerts.

The culprit? Someone in the business team added user_id and request_id as Prometheus labels. Classic cardinality bomb.

I’ve been through this rodeo enough times to know: high cardinality isn’t a bug, it’s a design failure. But the scary part is how silently it creeps up on you. One day you’re at 100k series, the next you’re at 2 million, and OOM is knocking on the door.

Symptoms: How to Know You’re Screwed

Before you start debugging, here’s what high cardinality looks like in the wild:

1. Prometheus memory usage keeps climbing—the head block inflates in RAM, GC pressure spikes, CPU follows.
2. /api/v1/query response times drop from milliseconds to seconds—or return context deadline exceeded.
3. WAL writes slow down—every new label combination creates a new series, and disk I/O goes through the roof.
4. Grafana dashboards fail to load—not Grafana’s fault, Prometheus just can’t serve the data.
5. The most obvious indicator: prometheus_tsdb_head_series is way higher than you expect. You budgeted for 100k, you’re seeing 2 million.

As someone on Reddit put it: “Timeseries data with high cardinality is always a pain point.” Truer words.

Root Cause: Who’s Planting the Bombs?

High cardinality has one root cause—uncontrolled label value combinations.

Here’s the textbook example:

http_requests_total{path="/api/v1/users", user_id="12345", status="200", method="GET"}

If you have 1 million users, that single metric spawns 1 million time series. Throw in path, status, and method combinations, and you’re looking at tens of millions. Prometheus TSDB was designed for “many time series with few data points each,” not “infinite series count.”

Which labels are most dangerous?

There’s a neat open-source library called promcap (MIT license) that caps cardinality at the source. The idea is simple—limit label combinations at registration time, overflow gets dropped or aggregated. Someone on Reddit recommended it, saying it “capped Prometheus metric cardinality at the source.” I like this approach better than cleaning up after the fact.

Step 1: Confirm the Damage—Check Total Series Count

Stop guessing. Check the numbers.

# Get current total time series count
curl -s http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_series | jq '.data.result[0].value[1]'

# If it's 2-3x your expected baseline, you have a problem

I set up a Grafana panel that polls this every minute. If the curve is linear growth instead of a flat line, something’s exploding.

Step 2: Find the Culprit—Which Metric Is the Worst Offender?

Prometheus has a debug endpoint that shows series count per metric.

# Top 10 metrics by series count
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName | to_entries | sort_by(.value) | reverse | .[0:10]'

Output looks like:

[
  {"key": "http_requests_total", "value": 523400},
  {"key": "api_latency_seconds", "value": 312000},
  {"key": "db_queries_total", "value": 89000}
]

If http_requests_total has 523k series, you’ve got a label explosion on your hands.

Step 3: Pinpoint the Exploding Label

Knowing the metric isn’t enough—you need to know which label is causing the explosion.

# Check label value distribution
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.labelValueCountByLabelName | to_entries | sort_by(.value) | reverse | .[0:10]'

If user_id shows 500k unique values and status shows 5, you found your killer.

Step 4: Emergency Triage—Kill the Label with Relabeling

Before you fix the root cause, make Prometheus viable again. Use relabel_config to drop the problematic label.

# prometheus.yml
scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['localhost:8080']
    metric_relabel_configs:
      # Option A: Drop user_id entirely
      - source_labels: [user_id]
        action: labeldrop
        regex: .*
      
      # Option B: Redact the value (gentler)
      # - source_labels: [user_id]
      #   action: replace
      #   target_label: user_id
      #   replacement: 'redacted'

Note that metric_relabel_configs runs after scraping but before storage. The metric survives, but the label is gone. You’ll lose the user_id dimension for queries, but that beats having Prometheus crash.

Step 5: Long-Term Fix—Plug the Leak at the Source

After the fire is out, you need to get the dev team to change their code. This is the hard part—it involves cross-team coordination.

Option 1: Application-Level Limits (Recommended)

Cap label combinations in code. If you’re using Go, promcap handles this:

import "github.com/your-org/promcap"

counter := promcap.NewCappedCounter(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests.",
    },
    []string{"path", "status", "method"},
    1000, // Max 1000 label combinations
)

After 1000 combinations, new ones get dropped or bucketed into __overflow__.

Option 2: Aggregate Before Reporting

Roll up high-cardinality dimensions before they hit Prometheus.

# Python example
def report_request(path, status, duration):
    # Aggregate /api/v1/users/12345 to /api/v1/users/{id}
    aggregated_path = re.sub(r'/users/\d+', '/users/{id}', path)
    METRIC.labels(
        path=aggregated_path,
        status=status
    ).observe(duration)

Option 3: Recording Rules for Pre-Aggregation

If historical data is already corrupted, use recording rules to reduce query-time pressure.

# rules.yml
groups:
  - name: cardinality_control
    rules:
      - record: job:http_requests_total:sum
        expr: sum(http_requests_total) by (job, status)

But this is a band-aid. The raw high-cardinality data still eats memory.

Step 6: Prevention—Set Hard Limits

Prometheus 2.30+ supports per-scrape sample limits:

scrape_configs:
  - job_name: 'my-app'
    sample_limit: 5000  # Max 5000 samples per scrape

Excess samples get dropped and trigger prometheus_target_scrapes_sample_out_of_order_total.

But sample_limit only limits samples, not series. For that, use label_limit:

scrape_configs:
  - job_name: 'my-app'
    label_limit: 50  # Max 50 unique labels per metric

Together, these two options catch most accidental explosions.

Architectural Rethink: When Single-Node Prometheus Isn’t Enough

If your business inherently requires high cardinality (multi-tenant SaaS, per-user metrics), no amount of tuning will save a single Prometheus instance. Time to think about architecture.

A Reddit user mentioned “shard your data, so every shard will contain only a subset of the labels.” That’s the ultimate fix—route different tenants to different Prometheus instances, each handling only its own label space.

FAQ

Q: Does high cardinality always cause OOM? A: Not always, but the probability is high. Prometheus TSDB keeps all active series metadata in memory. More series = more memory. At ~100k series, you’re looking at 2-4GB RAM depending on label count. Exceed physical memory, and OOM Killer steps in.

Q: How do I clean up high-cardinality data without restarting Prometheus? A: relabel_config changes take effect dynamically—no restart needed. You can also use promtool tsdb delete-series for historical cleanup, but back up WAL first.

Q: How badly does high cardinality affect query performance? A: Badly. A high-cardinality metric query might scan millions of series, pushing latency from milliseconds to seconds or minutes. Functions like rate() and increase() make it exponentially worse.

Q: Can I monitor cardinality with Grafana? A: Yes. Use prometheus_tsdb_head_series and prometheus_tsdb_head_chunks. Alert when head_series shows a steep upward slope.

Q: Why is user_id specifically a cardinality bomb? A: Because user_id values are unbounded. Each new user adds a new time series. 1 million users = 1 million series. Combine with other labels and you’re in the tens of millions. Prometheus wasn’t designed for this.

References & Community Insights

This article synthesizes knowledge from multiple community sources:

• Reddit r/opensource: The promcap library author shared how to cap cardinality at the source—practical and worth adopting.
• Hacker News: SREs shared real-world experiences with relabel tricks and architecture upgrades for high-cardinality scenarios.
• X/Twitter: Observability experts compared VictoriaMetrics and Thanos for high-cardinality workloads, with VictoriaMetrics winning on memory efficiency.
• Google Search Results: Multiple deep-dive blogs analyzed Prometheus TSDB memory models and cardinality impact mechanisms.

Special thanks to the promcap project author for sharing a genuinely useful open-source tool on Reddit.