What is high cardinality in Prometheus?

The definition, precisely

A Prometheus time series is the combination of a metric name and one specific assignment of values to every label that metric carries. Cardinality is the count of distinct combinations that actually occur. A metric with no labels has a cardinality of exactly one series per target, because the name alone is the identity. Add a label with 5 possible values and cardinality becomes 5 per target. Add a second label with 10 possible values and it is not 15 — it is 5 × 10 = 50, because every value of the first label can occur alongside every value of the second. This is a Cartesian product, and it is the entire reason cardinality problems sneak up on people: each label looks small in isolation, and the multiplication is invisible until someone adds it up.

“High” is doing informal work in the phrase “high cardinality” — Prometheus does not ship a single numeric threshold that flips a metric from fine to dangerous. Prometheus's own instrumentation guidance instead gives a concrete guideline for individual labels: keep a label's cardinality below 10 as a general rule, treat anything over 100 as worth investigating alternatives for, and reserves “only a small number of metrics” for exceeding that. That is a rule about one label's own value count, not about the total series a metric produces once several labels are multiplied together — which is exactly the gap the calculator above is built to close.

Bounded versus unbounded labels

The practical distinction that matters more than any specific number is whether a label's possible values are bounded — a fixed, small, knowable-in-advance set — or unbounded — a set that grows with your business, your users, or your traffic. Prometheus's own naming and labels documentation states the rule directly: “Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.”

Usually fine (bounded)	Usually a mistake (unbounded)
HTTP method (GET, POST, …)	Raw request path with an embedded ID
A fixed list of environments or regions	User ID, account ID, session ID
A handful of status-code buckets (2xx/4xx/5xx)	Full email address
Pod/instance name, if it is genuinely small and stable	Pod/instance name in an autoscaling cluster that churns constantly
A product's fixed set of plan tiers	A raw error message string

Notice pod name appears on both sides. It is bounded, and therefore fine, in a small, stable cluster. It becomes effectively unbounded — and starts contributing to churn rather than a simple flat series count — the moment autoscaling or frequent rolling deploys mean new pod names are constantly being created and old ones retired. That is the same distinction the churn multiplier in the calculator above is modelling: the label is bounded in size at any one instant, but unbounded across the whole retention window.

Why it costs what it costs

Two mechanisms, both stated directly in Prometheus's own documentation rather than folklore:

• Memory. Recent samples for every active series sit in Prometheus's in-memory head block before they are flushed to disk. “Each labelset is an additional time series that has RAM, CPU, disk, and network costs,” and that cost is paid per series, regardless of how rarely any individual series is actually updated.
• Query time. A PromQL selector has to locate every matching series before it can aggregate them. The same query against a metric with 100,000 series does substantially more work than against one with 100, even when the two queries return an identically-shaped answer.

Prometheus's own documentation walks through a concrete illustration worth repeating exactly: a per-user quota metric instrumented across 10,000 nodes for 10,000 possible users reaches, in its words, “a double digit number of millions” of series, which it states directly “is too much for the current implementation of Prometheus.” That is not a worst-case hypothetical from a blog post; it is the project's own documentation naming the exact shape of the mistake this page is about.

A worked before/after

Take a metric with three ordinary, bounded labels: method (5 values), status (6 values), route (20 values), scraped from 10 targets. 5 × 6 × 20 = 600 combinations, ×10 targets = 6,000 series. Ordinary, healthy, nothing to flag.

Now someone adds a fourth label, user_id, to help trace one customer's slow requests, with roughly 50,000 distinct users. The label product becomes 600 × 50,000 = 30,000,000, and with the same 10 targets that is 300,000,000 series — a 50,000× increase from one added label. At a 15-second scrape interval that is 20 million samples every second, a rate no single Prometheus instance ingests; in practice the process runs out of head-block memory and is killed by the operating system long before disk becomes the visible symptom. Load this exact scenario with the “Load worked example” and “Add a user_id label” buttons above and watch the series count update the instant the label is added.

Common mistakes beyond user_id

user_id is the canonical example because it is the easiest to explain, but it is far from the only label that looks reasonable while being effectively unbounded. A few that recur across real incident reports:

• Raw request paths or full URLs. /users/48213/orders/9981 as a label value creates a new series for every unique ID that ever appears in a path, which for a busy API can mean an unbounded number of series accumulating for the lifetime of the metric. The fix is a route template — /users/:id/orders/:id — captured before it reaches the label, which is exactly what well-behaved HTTP instrumentation libraries do by default.
• IP addresses. Client or peer IP as a label looks bounded (there are only so many IPv4 addresses) but behaves unbounded in practice for any public-facing service, and doubly so behind NAT or a CDN where the address rotates constantly for reasons unrelated to your traffic.
• Timestamps or other continuously-varying values used as labels. A label whose value is a request timestamp, a trace ID, or any other field intended to be unique guarantees a new series on every single occurrence — the opposite of what a label is for. That data belongs in logs or traces, which are built to index high-cardinality, per-event data; Prometheus is built to aggregate a bounded number of series over time.
• Raw error messages or exception text. Exception messages routinely embed a variable detail — a file path, a byte count, a specific key that was missing — making each occurrence a distinct label value. A bounded error_type or error_code label captures the useful signal (which failure category, how often) without the unbounded tail.
• Kubernetes pod names, container IDs, or git commit SHAs. Each is bounded at any single instant — a cluster has a specific, countable number of running pods — but unbounded across the retention window in any environment with autoscaling, frequent restarts, or continuous deployment. This is the churn case discussed in the storage-sizing guide: the label itself is not the mistake, the assumption that it stays stable for the length of your retention window is.

Finding the label responsible

When a Prometheus instance is under memory pressure from cardinality, the fastest diagnosis path is Prometheus's own data, queried against itself:

1. Find which metric names contribute the most series: a query like topk(10, count by (__name__)({__name__=~".+"})) ranks metrics by their own series count.
2. For the worst offender, check each of its labels in turn with count by (label_name)(metric_name). The label whose count is closest to the metric's total series count is usually the one responsible.
3. Ask whether that label's value set is bounded or unbounded, using the table above as a starting point.

Once identified, the label can be dropped at scrape time with a metric_relabel_configs entry using action: labeldrop, which stops it from being stored going forward. The more durable fix is removing it from the instrumentation call itself, so it is never emitted in the first place — a relabel rule is a mitigation applied after the fact, not a substitute for not creating the problem.

A quick gut-check before you add a label

Four questions, asked before instrumentation ships rather than after an incident, catch most of the cases above:

1. Could I write down every possible value of this label right now? If the answer is a short, finite list — HTTP methods, a handful of environments — it is almost certainly safe. If the answer is “no, it depends on how many X we have,” treat it as unbounded until proven otherwise.
2. Does the value set grow with usage, or is it fixed by design? A label whose cardinality increases every time a new user signs up, a new order is placed, or a new pod starts is growing with your business, not with your instrumentation choices — and will keep growing for as long as the business does.
3. Would I ever query or alert on one specific value of this label? If the honest answer is “no, I'd only ever aggregate across all of them,” the label is not buying you anything a log line couldn't, at a fraction of the cost.
4. What happens to the number above if this ships to every replica in production, not just my local test? The multiplication in the calculator above is exactly this question, answered with your actual target count instead of the one running on a laptop.

Frequently asked questions

Is cardinality about the number of metrics, or something else?

Something else, and this is the single most common mix-up. Cardinality is the number of distinct time series a metric produces, which is a function of its label values, not a count of how many differently-named metrics you have instrumented. A codebase can have only twenty metric names and still have appalling cardinality, if even one of those twenty carries an unbounded label. Conversely, a codebase with two hundred metric names, each with no labels or only tightly bounded ones, can have entirely reasonable total series.

How do I find which label is causing the problem?

Prometheus exposes this directly: query count by (__name__)({__name__=~".+"}) sorted descending to find which metric names contribute the most series, then for a specific metric run count by (label_name)(metric_name) for each of its labels in turn — the label whose count is closest to the metric's total series count is usually the culprit, especially if that count looks like it scales with something external (users, requests, IDs) rather than something internal and bounded (environments, regions, a fixed set of status codes).

Can I just drop a label after the fact to fix it?

Yes, using metric_relabel_configs with an action of labeldrop in the scrape config — this happens after the label already reached Prometheus but before it is stored, so the incoming series are still identified with the extra label right up until the drop, meaning if the label's value differs across scrapes for what should be the same series, dropping it after the fact can merge distinct incoming series into one, which is usually the desired outcome for exactly this kind of cleanup but is worth confirming for your specific case. The more durable fix is removing the label from the instrumentation itself, so it is never emitted at all.

Does high cardinality on one metric affect unrelated metrics?

Yes, because cardinality cost is mostly a property of the whole Prometheus process, not siloed per metric. Head-block memory is shared across every active series regardless of which metric they belong to, so a single runaway metric can push the process into memory pressure that slows down or destabilises ingestion and querying for everything else that instance handles, including metrics with no cardinality problem of their own.

Is there a hard limit Prometheus enforces?

Not a built-in hard cap on total series by default, though operators commonly configure sample_limit per scrape job to reject a target's entire scrape if it exposes more series than expected, which turns a slow cardinality leak into an immediate, visible failure rather than a silent, gradual one. That is a blunt instrument — the whole scrape is dropped, not just the offending metric — but it is often preferable to discovering the problem only once memory pressure has already started.

More on this topic

• The full cardinality & storage estimatorAdd your own metrics and see active series, samples/sec and disk together
• How much disk does Prometheus actually need?The storage formula, retention trade-offs, and what bytes-per-sample really covers
• About this toolHow the maths is built and cross-checked

Related tools

• Cron expression generatorBuild and explain the schedule for a scrape config or recording rule
• JSON formatterFormat and explore JSON — useful for Prometheus API responses and Alertmanager configs

See all the tools on this site →