The most fun telemetry problems I've seen usually start with an incident where we could not see enough, so we added the missing field after the postmortem. Then that just-in-case feeling kicked in and we added the neighbouring fields too, because nobody wanted to go blind the next time. The decision was reasonable at the time, which is what makes the mess harder to catch.
After enough incidents you start carrying every scar with you: debug logs from old failures, labels added for one investigation, dashboards built during a rollout, alerts created after one bad night. Nobody who added any of it thought they were creating an observability problem; they thought they were making things easier for the next oncall engineer.
Then the bill rises, queries slow down, dashboards contradict each other, and security finds customer identifiers in places they should never have reached. Eventually you clean up a few fields, reduce retention, delete the stale dashboards, and sample more aggressively. Then the next incident happens and you start again.
Broken Image Observability
Telemetry Is Treated Like Exhaust
Telemetry often begins as output nobody designed. Logs, metrics, traces, profiles, events, and audit records come out the side of the application.
A database schema, an API contract, a queue, a cache, or a new external dependency will usually get reviewed, and security-sensitive product data gets some kind of review too. Telemetry changes often slide through as implementation detail.
In production, telemetry consumes CPU, memory, network, and disk. It draws engineering attention, security review time, and budget. It can get heavy enough to interfere with the workload it is supposed to reflect.
An application container gets sized for business logic, then an agent, sidecar, collector, logger, or profiler joins the party. Maybe the overhead is tiny per pod, but it can get enormous across a fleet. It's easy to miss when each team only sees its own service.
The Bill Is the Architecture Review You Skipped
A telemetry bill makes those small costs hard to ignore: too many clever components, too many retries, health checks, labels, and debug logs. Each looked harmless in isolation. The people adding them often don't see what they cost until much later.
A developer adds a field today, the reviewer sees useful context, the platform team sees an ingestion spike later, finance sees the invoice after that, and security finds the accidental data exposure during a review months later. By then the developer may have moved teams and service ownership changed.
That local decision became a global cost. The person adding the field doesn't necessarily own the ingestion budget, and the person who understands the risk may not have the authority to block it. The invoice finally gives someone a reason to ask why we're collecting all this.
Cardinality and Cost
The most common explanation of telemetry cost is cardinality. In a time series database, a metric becomes the combination of its name and its labels. Every unique label set creates a distinct time series.
This is fine when the labels are bounded: service, env, region, status_code, route_template, team, zone. These labels describe stable operational dimensions. They let you group, filter, alert, and compare without creating an unbounded mess.
Then a team adds user_id to a metric to track down a hot partition issue. Three weeks later, the platform team sees active series explode, the team's manager is told they got a huge bill, and security realizes customer identifiers are now part of metric storage.
The label was useful for the investigation. Keeping it afterwards is a separate decision, but by then nobody is reviewing it.
Logs Are Anxiety With Timestamps
Metrics usually explode through cardinality. Logs usually explode through fear. That fear is rational. Missing one log line during an incident can waste hours. So teams learn the easy lesson and add more logs.
If we never ask what a log line is for, we end up dumping everything. A good log explains what happened: a state transition or boundary crossing, a decision, a rejection, a fallback, or a failure.
Compliance makes this worse when retention and indexing get treated as the same decision. Maybe you need to keep audit records for ninety days. That does not mean every debug line belongs in a hot searchable index for ninety days.
The Signal Has to Support a Decision
Most telemetry cost discussions start with volume: gigabytes per day, spans per second, active series, indexed logs, retention days, cardinality, query load. I'm all for tracking those numbers, but I'd first ask what we're using the signals for.
A metric might drive an alert, a log might explain a state transition, and a trace might help diagnose a customer-visible path. Audit records have compliance uses; other fields help security investigate abuse. A dashboard can help someone decide whether to continue a rollout, or an attribute can help them compare normal and abnormal system behaviour. Whoever adds the signal should be able to explain how they expect to use it.
Take two Spark signals. The first is spark_job_task_failures_total, labelled by failure_type, stage, and application. It drives an alert. When it spikes, the oncall engineer can tell whether jobs are dying from OOM errors, shuffle fetch failures, executor loss, or timeouts. It changes the next move, so it earns its place.
The second is an executor heartbeat metric loaded with executor_id, application_name, user_id, and a generated run identifier, added on the fly for deeper visibility. I understand why someone would add it during an investigation. But now it has no alert, no runbook, and one abandoned dashboard, while new executors and runs keep adding series.
This is why I think teams forget YAGNI entirely when it comes to telemetry. We recognise speculative code as technical debt, but we're much more forgiving of speculative telemetry. By the time someone questions the bill, the team that added it may no longer own the service.
Tooling Does Not Decide Who Can Say No
Tooling is the easy argument because it feels technical. Datadog or Grafana. SaaS or self-hosted. OpenTelemetry or vendor agents. But tools do not answer the harder question: when should we add telemetry and who is allowed to say no?
Who approves an unbounded label? Who decides whether customer_id belongs in a metric, a trace attribute, a structured log, or nowhere at all? Who can force debug logs to expire after seven days? Who owns a dashboard after the incident that created it?
Agreeing on who can make those calls is political work. Buying a different tool won't settle it.
Give Teams Defaults and Someone to Ask
A new metric dimension should feel closer to a database migration than a logging tweak. Having the platform team review every metric pull request would create its own problems, though. Teams might avoid useful telemetry because the process feels heavy, or reviewers might start rubber-stamping approvals just to keep work moving. We'd slow delivery without getting anyone to think harder about what they're adding.
Give teams standard libraries and approved metric dimensions such as service, env, region, team, zone, status_code, and route_template. CI can reject user_id, session_id, trace_id, container IDs, and generated run IDs when used as metric labels. The collector can strip unsafe attributes, and the gateway can drop health checks and debug noise before they reach the vendor. Teams still need someone to review exceptions, including a new global dimension, a sensitive identifier, a new exporter, or a longer retention class.
Retention needs the same discipline. Every signal should be born with a life expectancy. Alert-driving metrics have longer retention. Diagnostic signals get a shorter hot window. Debug signals expire fast. Anything unclassified is short-lived by default. If a metric does not support an alert, dashboard, runbook, SLO, rollout check, compliance workflow, or security investigation, it should not live forever.
The platform team owns the paved road, the denylist, the collector rules, the exception process, and the budget boundaries. Service teams own the signals they emit and the decisions those signals support.
A Broken Mirror Is Worse Than No Mirror
A useless signal is easy to block at the collector. After six months, though, it has a dashboard, a half-dead alert, and one scary incident story everyone uses to protect it. Bad telemetry does not stay harmless for long. Soon, people treat it like something that must exist.
When telemetry is missing, you know there's something you can't see. Bad telemetry can be harder to question because there's a dashboard right there, giving you an answer. People trust it and act on it. That's how the mirror becomes part of the machine.
I want teams to be able to remove a signal when it has outlived its usefulness, without waiting for another incident review. That takes an owner who can make the call, including when the team that created it is gone. Otherwise, we'll keep paying for it because nobody wants to be the person who turned it off.