We're Missing a Dimension, So Add user_id as a Label?
Scenario
During the last incident, the question "which customers are hitting these errors?" could only be answered by grepping raw logs under pressure — slow, manual, and painful. The team's metrics are clean and bounded today: labels like route, status, and region. But the Prometheus server is already under memory pressure and has OOM-restarted before. Logs and a tracing system exist, though in practice nobody has invested in them.
In the retro, a teammate proposes the one-line remedy: "Just add user_id and request_id as labels on the request metrics. Then every dashboard can slice by user, and next incident we'll see the affected customers instantly."
The Quick Fix on the Table
Add user_id (and request_id) as Prometheus labels. It really is a one-line instrumentation change, and it appears to give exactly the per-customer visibility the incident lacked.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- Every label value combination is a separate time series. Prometheus keeps an in-memory entry per active series. A histogram with 12 buckets across a few routes and statuses, multiplied by tens of thousands of user IDs, turns hundreds of series into tens of millions. On a server that already OOMs, this isn't a risk — it's a scheduled outage of the monitoring system itself, most likely during the next incident.
request_idis the pathological case. A label whose value is unique per event creates one series per request — each scraped once and then dead. That is the textbook definition of what a time-series database cannot do; it is logging, implemented in the most expensive way possible.- Everything gets slower, not just the new queries. Cardinality bloats the index that every query touches. Dashboards on plain
route/statusaggregations degrade, rule evaluations lag, and alerting latency grows — the blast radius is the whole observability stack, not one metric. - The unbounded label is a time bomb, not a cost. A cost you can budget for;
user_idcardinality grows with your success and spikes with abuse (a credential-stuffing attack means a million new label values in an hour). You cannot capacity-plan for a label you don't control. - It wouldn't even answer the incident question well. Metrics are aggregates for detection and trends. "Which customers are affected and what did their requests look like?" is an investigation question about individual events — logs and traces are not a fallback for this, they are the correct tool.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Keep metric labels bounded by design, and write the rule down. Adopt an explicit instrumentation policy: labels must have known, low, org-controlled cardinality (
route,status,region,tenant_tier— yes;user_id,request_id,session_id, raw URLs — never). - Bridge metrics to traces with exemplars. Enable exemplar support (
--enable-feature=exemplar-storage) and attach trace IDs to histogram observations. During an incident, you click from the error-rate spike on the dashboard straight to concrete guilty traces — per-request detail without a single extra series. - Put user context in structured logs, where it belongs. Emit
user_id,request_id,trace_id, route, and status as fields in structured (JSON) request logs. In Loki/Elasticsearch-class systems, a query likesum by (user_id)over error logs answers "which customers?" in seconds — the exact query the incident needed, on a system built for unbounded dimensions. - Use traces for the request-shaped questions.
request_id-style investigation is what tracing exists for: put user and request attributes on spans, use tail-based sampling to keep all error traces, and search by attribute in the tracing backend. - If a bounded slice of "user" is genuinely needed in metrics, aggregate it. A
tenant_tieror top-N-accounts recording rule, or a boundedcustomer_segmentlabel, often satisfies the dashboard need at a cardinality you can count on one hand. - Defend the TSDB you already have. Audit current cardinality (
topk(10, count by (__name__)({__name__=~".+"})), or the/tsdb-statuspage), setsample_limit/target limits on scrape configs so one bad deploy can't melt Prometheus, and alert on series-count growth. - Close the actual incident gap with a drill. Wire the dashboard-to-exemplar-to-trace-to-logs path end to end, then run the last incident's question through it as a game day. The gap was the workflow, not a missing label.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Each label value combination is its own time series held in memory. user_id turns hundreds of series into millions on a Prometheus that already OOMs — we'd be scheduling our monitoring to die during the next incident, which is the one time we need it."
- "request_id is one series per request, scraped once and abandoned. That's logging implemented inside a TSDB at the worst possible price."
- "The incident question was 'which customers are affected' — that's an investigation over individual events, and structured logs answer it in one query. Metrics are for detecting that something is wrong, not for enumerating who."
- "Exemplars give us the click-through we actually wanted: from the error spike on the dashboard straight to real traces with user IDs on them — zero additional cardinality."
- "It's not one small change, because cardinality isn't a one-time cost — it grows with our user count and spikes with abuse traffic. A label we don't control is a label we can't capacity-plan."