Most teams discover they need observability on the worst possible day: the checkout page is slow, dashboards are green, and nobody can say which service is actually on fire. That gap between "we have monitoring" and "we can explain what the system is doing" is the whole discipline in one sentence.
Monitoring answers questions you already knew to ask — is the CPU high, is the endpoint up. Observability lets you ask a new question of production without shipping new code. OpenTelemetry, Prometheus, Grafana, and distributed tracing are the tools; the skill is deciding what to record, what to alert on, and what to ignore.
What observability actually is (and why the tooling alone does not give it)
Three signal types carry almost all the weight: metrics (aggregated numbers over time), logs (discrete events with context), and traces (one request's path across services). Each answers a different question, and teams usually over-invest in one.
Metrics are cheap to store and expensive to interpret — a p99 latency line tells you something is wrong but not where. Logs are rich but unsearchable at scale unless you control cardinality. Traces are the only signal that shows causality across service boundaries, which is why distributed tracing is the piece most self-taught engineers skip and later regret.
The non-obvious part: observability is a data-modelling problem before it is an infrastructure problem. If your trace lacks a service name, a status code, or a request ID that also appears in logs, no amount of Grafana will connect the dots. OpenTelemetry exists precisely because vendor-specific agents made that connection fragile — the SDK emits a vendor-neutral format, and the backend (Jaeger, Tempo, a commercial APM) becomes a swappable detail.
One walkthrough: from "users say it's slow" to a root cause
Imagine a checkout flow: frontend → API gateway → orders service → payments service → a Postgres instance. Users report slow checkouts. Here is how a practitioner actually works, in order.
- Start from the symptom, not the dashboard. Pull the p95 and p99 latency for the checkout endpoint over the last 24 hours. Say p50 is 120 ms and p99 is 2.4 s — the average hides the problem, and the tail is what users feel.
- Break it down by dimension. Group the same metric by status code and by downstream dependency. If p99 spikes only for requests hitting the payments service, the search space just shrank by 80%.
- Open a trace. Find a slow trace and read the span waterfall. The pattern to look for is a single span that dominates the total — a 2.1 s payments call inside a 2.3 s request is a smoking gun.
- Correlate with logs. Take the trace ID from that span, search logs for it, and check whether the payments service logged retries, connection-pool exhaustion, or a timeout just under its client deadline.
- Confirm with a counter. Add or read a Prometheus metric for pool saturation or retry count. If retries climb in lockstep with p99, you have a mechanism, not a correlation. For a deeper dive into this topic, see SQL Mastery (Window Functions, Indexes).
The failure mode of doing it differently: teams start at step 5 (stare at host CPU), find nothing, and conclude "it's the network." Host metrics are almost always flat when the problem is a slow dependency or a retry storm.
Where the scheme itself breaks
Observability designs fail in predictable ways. These are not beginner mistakes — they are structural limits.
Cardinality explosion. Every unique label combination in Prometheus is a separate time series. A label like user_id or request_id on a metric will multiply series until the scrape target falls over. It looks fine in staging with ten users. Early sign: scrape duration climbing and memory on the Prometheus pod growing steadily. If it is already too late, drop the offending label, aggregate, and move high-cardinality data to traces or logs where it belongs.
Sampling that hides the tail. Head-based sampling at 1% is fine for cost and useless for debugging rare errors — the slow requests you care about are exactly the ones not sampled. Early sign: you can see errors in aggregate but never find a matching trace. The fix is tail-based sampling: keep every trace with an error or above a latency threshold, sample the boring ones.
Alerts that fire on causes, not symptoms. CPU at 90% is not an incident; a failing SLI is. Symptom-based alerting (error rate, latency against an SLO) pages less and means more. Early sign: on-call ignores pages because "it's just that alert again." By then the alerting system has lost credibility and needs a rebuild, not a tweak.
Blackbox silence. Everything inside the cluster is instrumented, but nobody checks the thing users actually hit. A load balancer returning 502s with healthy backends shows up in no internal metric. Cheap external probes against the public endpoint catch this class of failure.
| Failure | Early signal | Recovery |
|---|---|---|
| Cardinality explosion | Scrape duration and memory rising | Drop labels, aggregate, move to traces |
| Head sampling hides tail | Errors visible, traces missing | Switch to tail-based sampling |
| Cause-based alerts | On-call ignores pages | Rebuild around SLIs and SLOs |
| No blackbox checks | Internal green, users red | External probes on public endpoints |
How people actually learn this
Self-study works if you already run something in production. You read the OpenTelemetry docs, stand up Prometheus and Grafana in Docker, and learn by breaking things — the cost is time and the fact that nobody tells you your SLO is meaningless or your alert is noise.
Courses compress the feedback loop: someone has already made the cardinality mistake and can show you the shape of it before you repeat it. The trade-off is real — you still need your own hands on a running system to internalize it. Bootcamps that promise production readiness in a weekend do not; observability is judgment, and judgment comes from incidents. For a deeper dive into this topic, see CI/CD as an Art: A Real Pipeline Breakdown and Costly Mistakes.
The middle path most engineers take: build a small instrumented stack yourself, then get structured feedback on the decisions — sampling strategy, alert thresholds, runbook structure — because those are the parts where a wrong choice is invisible until an incident.
Who this suits and who it does not
It suits backend and SRE-adjacent engineers who own a service in production, platform engineers building shared telemetry pipelines, and on-call responders tired of guessing. It does not suit people who want a checklist to install tools — the tools are the easy part, and the discipline is deciding what to measure and what to page on.
If you are not yet running something with real users, start there first; observability without a system to observe is theory.
If you want a structured path through the decisions — SLI/SLO design, OpenTelemetry instrumentation, Prometheus and Grafana, tracing, alerting, runbooks and postmortems — the Observability course walks through exactly these trade-offs with worked examples and gives feedback on your choices. Start by instrumenting one real endpoint, then follow the path from there.