Strategies to reduce Prometheus metric cardinality

Prometheus cardinality is the number of observed metric-name and label-set combinations your server holds; multiplying each label’s distinct-value count gives the maximum when every possible combination exists. Every active series lives in memory, Prometheus has no built-in hard cap on series count, and when the count outgrows the host, the kernel OOM killer terminates the process. Diagnose which metrics and labels drive the count, then apply relabeling, recording rules, hard limits, and bounded label design to reduce Prometheus cardinality without losing the signal you alert on.

What Prometheus cardinality is and why it explodes

Start with the math, because the math is what makes cardinality multiplicative rather than additive. A time series is a metric name plus one unique combination of label key-value pairs. A metric with a method label containing 4 values and a status label containing 5 values, plus a path label containing 100 values, can produce up to 2,000 series. Add one more label with 50 values and you are at 100,000. The maximum series count is the product, not the sum, of label value counts.

The product stays manageable as long as every label is bounded. It explodes when any label is unbounded: user_id, session_id, request_id, url_path, container_id, any label whose value set grows without a fixed ceiling. Datadog’s custom metrics billing treats each unique combination of metric name and tag values as a separate billable timeseries, which is why these same labels show up as the canonical causes of surprise bills.

Kubernetes exposes high-churn identity metadata by design. Service discovery provides pod names and container IDs as __meta_* labels, but Prometheus removes labels beginning with __ after target relabeling unless you explicitly retain them. Exporters such as kube-state-metrics expose separate pod and ReplicaSet identity labels, and those persisted values change with rollouts or restarts, so deployments can mint fresh series while older series linger in the head block.

How high cardinality hurts memory and performance

That multiplicative growth first lands in memory. Prometheus keeps all active series in the TSDB head block in RAM. Pipedrive hit a failure threshold around 8 million active series on a 32 vCPU, 256 GB instance and reported that adding more resources beyond that point was futile.

The second cost is query latency. PromCon 2019 measurements put an instant count(metric_a) query at ~1.5 seconds against 100,000 matched series and ~5 seconds against 200,000, with the presenter noting that queries on a single metric “begin to be painfully slow above ~100,000 series.” Dashboards built at one scale degrade quietly as series counts climb.

Storage and egress add another cost: more series means more local TSDB blocks and more remote write volume, which turns directly into money on any managed backend billed per series or per sample. The worst case combines memory pressure and slow queries while storage volume rises. Trivago’s postmortem describes 3,000+ unique pods in a 3-hour window driving “100% memory saturation on some nodes and out-of-memory events that killed Prometheus processes.” Pipedrive also reported WAL replay taking up to 15 minutes after a restart, creating 15 minutes of monitoring blindness during an incident.

Diagnosing cardinality: TSDB status page and PromQL

Before you cut anything, measure which metrics are largest, because the offenders are rarely what you assume. The cheapest tool is built in: the TSDB status page in the Prometheus UI, backed by GET /api/v1/status/tsdb. It ranks the top label names by distinct value count, the top metric names by series count, the top label names by memory usage, and the top label=value pairs by series count. Grafana troubleshooting guidance offers a useful threshold: any label with more than 10K unique values is almost certainly a bug, and a histogram _bucket metric at the top of the series-count table is almost always the answer.

For trend data, query prometheus_tsdb_head_series, the gauge exposing current in-memory series count. To rank offenders per job, two queries from the OKD monitoring docs do most of the work:

topk(10, max by(namespace, job) (topk by(namespace, job) (1, scrape_samples_post_metric_relabeling)))

topk(10, sum by(namespace, job) (sum_over_time(scrape_series_added[1h])))

Use the diagnostic tools as an ordered workflow:

1. Compare the two PromQL queries. The first ranks jobs by post-relabeling sample count, while the second ranks jobs by new series created over the last hour and surfaces churn.
2. Inspect the TSDB status page to identify the metric names and label values behind the largest jobs.
3. Run promtool tsdb analyze for offline churn and cardinality analysis against TSDB data.
4. Profile the heap through the pprof endpoint when you need to identify the structures consuming memory. Prometheus developers wrote the server in Go, so pprof provides a heap-level breakdown.

Coveo’s investigation used that profiling workflow to trace an OOM to kubelet container metrics whose id label carried 116,525 distinct values; the fix cut sample rate 75% and dropped pod memory from a 30 Gi ceiling to 8 GB.

Dropping labels at scrape time with metric_relabel_configs

Once diagnosis names a guilty label, the first remediation is to strip it before it ever reaches storage. metric_relabel_configs runs after the scrape and before ingestion, so a labeldrop there removes the label from every series in that job:

scrape_configs:
  - job_name: kubelet
    metric_relabel_configs:
      - action: labeldrop
        regex: 'path|replicaset|controller_revision_hash|owner_name'

That exact rule comes from a production incident where dropping high-churn labels cut a Prometheus instance from ~10M series and ~60 GB of memory to under 4M series and 18–25 GB. labelkeep is the inverse: it keeps only labels matching the regex and drops everything else, which is safer when you can enumerate what you need.

One risk deserves attention before you ship this config. If the dropped label was the only thing differentiating two series, they collapse into one. Prometheus keeps only the first sample per timestamp and never merges or sums duplicate samples.

Since Prometheus 2.52.0, Prometheus increments prometheus_target_scrapes_sample_duplicate_timestamp_total when it detects duplicate series during one scrape, as documented in the duplicate-sample metric; earlier versions lost the duplicate silently. A documented implementation gap can miss collisions that arise only after metric relabeling, so validate the resulting label sets and watch the counter after any labeldrop change.

Dropping entire metric families before ingestion

Label surgery assumes the metric itself is worth keeping. Often it is not: a large share of ingested series never appears in any dashboard or alert. For those, drop the whole family at scrape time by matching on __name__:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'node_sockstat_.*|node_vmstat_.*|node_uname_info'
    action: drop

The hard part is knowing which families are safe to drop, and mimirtool analyze answers that empirically. The workflow extracts every metric referenced in dashboards, does the same for rules covering recording and alerting, then compares both lists against live active series. The output’s additional_metric_counts field lists metrics absent from all dashboards and rules, with per-job series counts, producing a direct drop-candidate list. One documented example contained 38,184 total active series. Queries used only 14,047 of them, leaving 24,137 series unused.

One scope limitation remains: mimirtool sees dashboards and rules, including alerts and recording rules, so it misses ad-hoc queries run through Grafana Explore or the API. Confirm with the teams who own the metrics before dropping.

Filtering at egress with remote_write and write_relabel_configs

Scrape-time relabeling controls what enters your local TSDB. If you also ship to long-term storage, write_relabel_configs under the remote_write stanza gives you a second, independent reduction layer: series you want locally for short-term debugging can still be filtered before they leave the box.

remote_write:
  - url: https://your-backend/api/v1/push
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'thanos_objstore_bucket_operation_duration_seconds_bucket'
        action: drop

This layer matters most when the destination bills on what arrives. Grafana Cloud charges $6.50 per 1,000 active series in its 10,001–100,000 band, and Grafana calculates the bill from the 95th percentile. AWS pricing lists $0.90 per 10 million samples ingested in the first tier for AWS Managed Service for Prometheus. Every series filtered at egress is a series the backend never meters. Grafana’s own cost documentation recommends sender-side write_relabel_configs over receiver-side filtering.

Pre-aggregating with recording rules instead of dropping

Dropping has a failure mode the previous three sections share: if an alert or dashboard references the dropped series, it breaks. Recording rules avoid that failure by pre-computing an aggregation and storing the result as a new, lower-cardinality series, so the pod-level detail disappears but the service-level signal survives.

The Prometheus documentation recommends aggregating with without rather than by: name the labels you are removing, and Prometheus preserves everything else, such as job. A hierarchical example from the official practices page:

- record: instance_path:requests:rate5m
  expr: rate(requests_total{job="myjob"}[5m])

- record: path:requests:rate5m
  expr: sum without (instance)(instance_path:requests:rate5m{job="myjob"})

The naming convention is level:metric:operations: the labels present in the output, the unchanged metric name, then the operations applied. Two rules from the same page keep aggregations statistically valid: when aggregating ratios, aggregate numerator and denominator separately and then divide. Never average a ratio or average an average. Repoint dashboards and alerts at the recorded series, then drop the raw high-cardinality series at remote write once nothing references it.

Setting hard limits for scrapes and targets

Relabeling and recording rules fix today’s cardinality; they do nothing about the instrumentation mistake someone ships next quarter. Prometheus has native guardrails for that. Every guardrail defaults to 0, which means unlimited, and you can configure each one globally or per scrape job:

• Scrape limits: sample_limit has existed since v1.5.0 and caps samples per scrape after metric relabeling. On breach, Prometheus fails the entire scrape, reports up=0 for the target, and rejects all samples from that scrape. label_limit has existed since v2.27.0 and caps labels per series after metric relabeling; exceeding it also fails the whole scrape. Companion settings label_name_length_limit and label_value_length_limit cap string lengths.
• Target limits: target_limit has existed since v2.21.0 and caps total scrape targets per job after target relabeling. On breach, Prometheus marks the over-limit target set as failed and does not scrape those targets.

The scrape-limit failure mode isolates a misbehaving target so its samples do not consume the whole server. Cloudflare’s guardrails include a default sample_limit of 200 per application across 916 Prometheus instances scraping 4.9 billion total series, precisely because a cardinality explosion means “you lose all observability as a result.” Pair the limits with the mixin alerts PrometheusScrapeSampleLimitHit and PrometheusLabelLimitHit so a breached limit pages someone instead of silently blanking a target.

Kubernetes and service mesh cardinality sources

Those guardrails matter most in the environments that generate cardinality fastest, and Kubernetes is the worst offender. The kube-prometheus-stack Helm chart enables kubelet with cAdvisor and kube-apiserver by default, alongside kube-state-metrics, node-exporter, and the control-plane components. Persisted identity labels from exporters among these targets churn with every rollout. The chart already ships default cAdvisor drop rules for known-noisy metrics, but the replicaset pattern still bites: one ReplicaSet incident saw a broken Keycloak deployment create ~36,000 ReplicaSets, adding ~60,000 head series and causing the kernel OOM killer to terminate Prometheus every four minutes despite a 6 Gi limit. The recurring fix across independent postmortems is dropping replicaset and controller_revision_hash labels, along with the kube_replicaset_* family.

Service meshes multiply on top of that Kubernetes churn. The istio_requests_total metric carries 20 labels, and both sidecars in every connection emit the metric, once as reporter=source and once as reporter=destination, doubling everything. One published estimate puts a 100-service mesh at roughly 355,000 time series across Istio’s standard counters and histograms.

The escape hatch is the pattern from the previous section applied at mesh scale: Karl Stoney’s federation cut 337,635 Istio series to 10,143, a 97% reduction, across 330 workloads. Istio’s observability best practices recommend the same recording-rule aggregation, and the Telemetry API can remove tags at generation time so unwanted labels are never emitted.

Lower-effort volume reductions

Not every fix requires relabeling surgery. Several adjustments cut volume with one configuration change each.

• Widen scrape_interval for low-priority targets. The default is 1m; moving a target from 15s to 60s cuts its sample rate 4×. Two constraints from Robust Perception guidance: stay at or under 2 minutes, because a single failed scrape at longer intervals runs into the 5-minute staleness lookback, and remember that rate() windows should be at least 4× the scrape interval.
• Disable unused node_exporter collectors. Version 1.12.1 enables 50 collectors by default. Some off-by-default collectors are notorious when enabled. On 32-vCPU machines, the interrupts collector accounted for ~3,900 of 7,500 total metrics per machine in an interrupts collector report. On NFS clusters, mountstats produced more than half of all stored metrics in a mountstats report. Use --no-collector.<name>, or --collector.disable-defaults plus an allow-list. Disabling at the exporter beats relabeling here because relabeling still pays the collection cost on the exporter side.
• Prune histogram le buckets. A classic histogram with the client_golang defaults produces 14 series per label combination: 12 _bucket series plus _sum and _count. A metric_relabel_configs rule dropping unneeded le values cuts that multiplier directly, at the cost of wider interpolation intervals for histogram_quantile, so keep boundaries near your SLO thresholds. For new instrumentation, native histograms became stable in Prometheus v3.8.0. The Prometheus project recommends native histograms over classic histograms because they remove the bucket-count trade-off.

Preventing cardinality at the source: label design

Every technique so far cleans up after bad labels; label design prevents them. The dividing line is whether a label’s value set has a fixed ceiling. HTTP method is a bounded enum: GET, POST, PUT, DELETE. user_id has no ceiling, and neither does a raw url_path that embeds IDs. Bounded labels multiply into a predictable product; one unbounded label makes the product unbounded.

The Prometheus FAQ’s blunt line “Don’t use Prometheus for logs!” extends naturally to high-cardinality identifiers. Put per-request identifiers such as user IDs and request IDs in traces or logs. Keep metric labels bounded. Traces handle per-event context well because a span carries arbitrary attributes without creating a standing series. OpenTelemetry, commonly shortened to OTel, supports custom business-logic attributes on spans through its SDK instrumentation. GoDaddy’s Istio postmortem shows the cost of getting the boundary wrong: GoDaddy reported that Istio interpreted unique Host headers as new service identities, so each user’s request added around 25 new time series to Prometheus.

Enforce the rule in code review: any new metric label must have an enumerable value set, and anything per-request goes on a span.

Alerting on cardinality growth before an outage

Design discipline still leaks, so you want an alarm that fires before the OOM does. Notably, the official prometheus-mixin ships no alert on prometheus_tsdb_head_series; no community-standard cardinality-growth alert exists, so you write your own. Grafana’s alerting examples give two starting expressions:

# Absolute ceiling for a single instance
prometheus_tsdb_head_series > 1.5e6

# Rapid growth: more than 1,000 series added in 10 minutes
delta(prometheus_tsdb_head_series[10m]) > 1000

A churn-focused variant catches series being created faster than they expire:

rate(prometheus_tsdb_head_series_created_total[5m]) - rate(prometheus_tsdb_head_series_removed_total[5m])

A persistently positive result means the head block is filling. Calibrate every threshold to your instance’s measured capacity; 1.5e6 and 1,000-per-10-minutes are documented examples, not universal constants. When the alert fires, the per-job scrape_series_added query from the diagnosis section tells you which job is responsible.

When relabeling is not enough

Sometimes the alert fires and the honest answer is that the cardinality is legitimate: large fleets and service meshes can produce per-tenant metrics that dashboards require. At that point you are fighting a single-instance TSDB ceiling, and the fixes are architectural.

• Thanos scales Prometheus horizontally over object storage, with per-tenant series limits available in Receive mode.
• Grafana Mimir: Grafana Labs reports that it tested Mimir at 1 billion active series in a single cluster. Mimir’s preferred ingest-storage architecture in version 3.0 and later uses Apache Kafka, while classic ring-based ingestion remains available. Mimir enforces per-tenant limits like max_global_series_per_user with a default of 150,000.
• VictoriaMetrics is a PromQL-compatible backend with proprietary columnar storage and node-level series limits such as -storage.maxHourlySeries.
• Federation shards scraping hierarchically across Prometheus instances. Federation powered the 97% Istio reduction cited earlier, but multi-cluster federation is operationally heavy.

Each architecture raises the memory ceiling. Long-term storage that charges per series or per sample still meters every retained series, so relabeling remains both a health control and a cost control.

groundcover takes a different architectural position on data access and residency. Bring Your Own Cloud (BYOC) is the default architecture across every tier: the data plane runs inside your own VPC, with VictoriaMetrics storing metrics and ClickHouse storing logs and traces alongside Kubernetes events. The remotely managed control plane handles the UI and orchestration, while telemetry storage and processing stay inside your infrastructure.

That split-plane architecture lets teams retain comprehensive observability data without routing telemetry through a vendor-hosted storage plane. Pricing remains flat per node rather than varying with series cardinality or data volume, so teams can prune metrics according to Prometheus memory and query health instead of rationing visibility to control ingestion. If you want to validate the architecture on your own infrastructure, the free tier runs the same default BYOC model on a single cluster, with no credit card required.

FAQ

How do I identify which metrics are driving cardinality? Open the TSDB status page, backed by /api/v1/status/tsdb, and read the ranked tables: top metrics by series count and top label=value pairs by series count. A _bucket histogram metric at the top, or any label with more than 10K unique values, is your likely offender. Track the total with prometheus_tsdb_head_series and rank jobs with scrape_samples_post_metric_relabeling.

What is the difference between labeldrop and labelkeep? labeldrop removes labels matching a regex and keeps everything else; labelkeep keeps only matching labels and removes the rest. labelkeep is safer when you can enumerate needed labels. Either way, verify that removing a label does not collapse distinct series into duplicates, and watch prometheus_target_scrapes_sample_duplicate_timestamp_total after the change.

When should I use write_relabel_configs instead of metric_relabel_configs? Use metric_relabel_configs to keep series out of your local TSDB entirely. Use write_relabel_configs when you want data locally for short-term debugging but filtered before it reaches metered long-term storage.

How do recording rules reduce cardinality without breaking dashboards? A recording rule stores an aggregation like sum without (instance)(...) as a new series, collapsing the high-cardinality dimension while preserving the aggregate. Repoint dashboards and alerts at the recorded series first, then drop the raw series at egress.

Do sample_limit and label_limit reduce my existing cardinality? No. Both default to 0, which means unlimited, and only prevent growth: Prometheus fails a scrape that exceeds either limit and reports up=0 for the target. These limits prevent future instrumentation mistakes. Use relabeling or aggregation to clean up existing cardinality.

What makes a label safe at the source? A fixed, enumerable value set. HTTP method qualifies; user IDs, request IDs, and raw URL paths do not. Put per-request identifiers on trace spans, where they add context without creating distinct metric series retained until their data ages out under the configured retention policy.

How do I alert on cardinality growth? Alert on prometheus_tsdb_head_series with both an absolute threshold and a short-window delta() for growth spikes, and monitor net churn via prometheus_tsdb_head_series_created_total minus prometheus_tsdb_head_series_removed_total. The official prometheus-mixin does not cover this, so calibrate your own thresholds to measured instance capacity.