1Understand
The idea
Observability is how you know whether the product is healthy for real users, not whether a process is running. Dashboards full of CPU graphs without a user journey and an SLO are ops wallpaper. In system-design interviews, observability is the bridge between “we built retries and replicas” and “we can prove checkout stays under 500ms p99 when payment is slow.”
You instrument metrics (aggregates over time), logs (discrete events with context), and traces (causal paths across services). Those three pillars feed SLIs, measurable proxies like success rate or latency, and SLOs, numeric targets on those SLIs. The error budget (100% minus SLO) is how mature teams decide whether Friday’s deploy is allowed: budget left, ship with canaries; budget burned, fix before new risk.
Interviewers reward RED (rate, errors, duration) on APIs, USE (utilization, saturation, errors) on databases and queues, and alerts that page on symptoms (SLO violation, payment error spike) not causes (one pod restarted). Tie every probe to a journey you already named in capacity planning, “add tracing on checkout” lands harder after you sized checkout QPS and latency budget.
Intuition
Start from the journey, not the toolkit. Pick one critical path, login, checkout, feed load, upload complete, and ask what “bad” means in user language: timeouts, wrong totals, stale UI, partial failure. That becomes your SLI list. Avoid measuring everything; measure what gates revenue or trust.
SLI → SLO → budget. An SLI is a measurement: successful_checkouts / total_checkout_attempts over a rolling window, or histogram http_request_duration_seconds for route=checkout. An SLO is a target: “99.95% checkout success per 30 days” and “p99 latency < 2s.” Error budget is the allowed un-good fraction, for 99.95% monthly availability, roughly 22 minutes of bad minutes-equivalent per month (order-of-magnitude; say you know it is approximate). Releases and experiments spend budget; postmortems explain spend.
Metrics (when to use). Counters and histograms for rates and percentiles; low cost at aggregate cardinality (route, status, not unbounded user_id on every series). RED per service: request rate, error ratio, duration percentiles. USE on resources: DB connection pool utilization, disk queue depth, Kafka consumer lag, saturation often predicts user errors before error rate jumps.
Logs. Structured JSON with trace_id, request_id, tenant_id (hashed if PII), level, stable field names. Logs answer “what happened for this id” after metrics/traces point at a spike. High-cardinality dimensions in log indexes (raw email per line) explode cost, aggregate in metrics, detail in sampled logs.
Traces. OpenTelemetry / traceparent propagation across gateway → app → DB → payment client. One slow checkout trace showing 1.8s in inventory retries beats grep across twelve unstructured log files. Sample 1-5% happy paths; always retain errors or tail p99.
Alerting philosophy. Page humans for user-visible SLO burn or multiplicative error rate; ticket for trends. “CPU > 70% for 1m” without checkout impact is noise. Multi-window burn rates (Google SRE book) catch fast and slow budget leaks. Runbooks link from alerts to dashboards and ownership.
What you will be able to do
- Define an SLI/SLO for a user journey
- Instrument latency, errors, and saturation
- Use traces for multi-service requests
When this comes up
- RED / USE metrics
- p99 latency
- Distributed tracing
- Error budget
2See it
Architecture sketch
Three pillars into alerts
Instrument the journey you sized in capacity planning.
Walkthrough
Checkout burns budget
Symptom alert beats cause spam
1 / 4
Checkout API emits rate, errors, duration (RED). Saturation shows on DB/queue (USE).
Read the walkthrough as text
Worked example, checkout SLO for a payments-integrated storefront.
Product: POST /checkout authorizes card, reserves inventory, sends confirmation email async. Interview asks: “How do you know checkout is healthy?”
- Name the journey SLIs (say assumptions): success = HTTP 2xx and
payment_status=authorized; latency = server-side time to response; exclude client aborts if policy says so. - SLO targets: “99.95% successful checkouts per rolling 30d; p99 latency < 2s for the sync path (email async).” Connect to business: missed SLO = lost sales + chargeback risk.
- Instrumentation: API middleware histogram
checkout_duration_secondswith buckets to 5s; countercheckout_outcomes{status=success|payment_decline|inventory_conflict|5xx}; propagatetrace_idinto payment HTTP client and inventory gRPC. - Saturation (USE): alert DB pool > 85% for 10m; payment worker queue lag > 30s; before user-visible 503s, interviewers like “leading indicators.”
- Error budget: dashboard shows budget remaining; deploy policy, canary 5% traffic; auto-halt if 5xx rate 2× baseline for 15m or fast burn alert fires.
- Incident: p99 jumps from 400ms → 1.2s; on-call opens trace sample, sees inventory client retry storm on 429; fix backoff, not “scale API pods.”
- Logs: correlate single failed
order_idviatrace_idacross services for support tickets.
What interviewers listen for: numeric SLO, not “Grafana”; symptom-based paging; metrics + traces + structured logs; explicit async boundary (email off critical SLI); you distinguish availability SLI from latency SLI.
Napkin math
SLI = successful checkouts / total checkouts
SLO = 99.9% over 30 days
error budget ≈ 0.1% ≈ 43 min downtime/month
alert on burn rate, not single pod restart
Metrics + logs + traces feed SLIs. Page on user symptoms.
3Answer it
How to answer in an interview
When prompts mention monitoring, on-call, reliability, or “how do you debug production”:
- Pick one user journey, checkout, signup, search, upload, match the product story.
- Define 1-2 SLIs, success rate, latency percentile, optionally freshness.
- State SLO numbers, monthly or weekly window; mention error budget in one sentence.
- Three pillars, which metric names/histograms, which log fields, trace propagation on critical hops.
- Saturation, pool, lag, disk, alert before pure error spikes.
- Alert + release policy, page on budget burn / user errors; canary tied to SLO.
- Debug story, “trace shows payment retry” beats “we would grep logs.”
Phrases to use: “symptom-based alerting,” “error budget,” “RED on the API, USE on the database,” “propagate traceparent,” “high-cardinality is expensive, aggregate in metrics.”
Answer checklist
Tradeoffs & numbers
- Tight SLO (99.99%): fewer outages tolerated; slower feature velocity; more redundancy cost, justify with revenue tier.
- Loose SLO (99%): cheap ops; users feel pain, OK only if product is internal or non-critical.
- High-cardinality labels (
user_idon every metric series): precise debugging, unsustainable cost, prefer exemplars or sampled traces. - 100% trace sampling: great forensics, 10-30% overhead, sample errors always; head-based 1-5% on success.
- Log everything: compliance vs dollars; retention tiers (7d hot, 90d cold).
- Synthetic probes: catch regional DNS/LB failures; miss real user device mix, combine with RUM if asked.
- When NOT to over-instrument: early MVP with 100 DAU, one golden-signal dashboard + basic alerts; skip full tracing mesh until scale demands it.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
API latency SLO
Example. Mobile home feed p99 must stay under 250ms monthly; spikes after deploy blocked the last release. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.
How. Histogram on `feed_handler_duration_seconds`; SLO on p99; canary compares canary vs baseline p99; trace slow spans to DB vs cache miss.
Saturation / capacity alerts
Example. Checkout errors rose only after connection pool exhausted for six minutes, CPU was idle. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.
How. USE metrics: pool active/limit, queue depth, Kafka lag; alert at 80% sustained; runbook scale replicas or fix connection leaks before user 503s.
Trace a slow request
Example. Support ticket: one user, checkout took 12s; logs alone show timeouts in three services. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.
How. Find `trace_id` from gateway access log; waterfall shows payment SDK retry loop; fix idempotent retries with cap, not add Redis “for speed.”
Error budget for releases
Example. Team wants Friday deploy; budget is 85% consumed after bad canary Tuesday. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.
How. Show burn rate math; freeze feature flags; ship only fix-forward; postmortem action items before next risky change. SLI → SLO → error budget; exemplar trace on slow checkout; page when budget burns, not on CPU noise.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
What you page on
Avoid. Page on every pod restart and CPU blip. Noise trains people to ignore pages.
Prefer. Page on user-journey SLOs and payment error spikes. Debug with traces.
Weak sketch
Bad
Stronger sketch
Good
No SLO
Why it happens. Dashboards without targets cannot rank incidents, justify headcount, or gate releases, on-call debates vibes at 3am.
How it fails. “We have Prometheus and Grafana on 400 panels” but no stated availability or latency goal for signup or checkout.
Fix. Pick one journey SLI; set SLO (e.g. 99.9% / 30d); track error budget; page on multi-window burn, not raw CPU.
Alert noise
Why it happens. Pager fatigue trains humans to ignore pages; real SLO breaches get lost in infra chatter. Metrics without SLO/error budget do not drive release decisions.
How it fails. Pages `#prod-alerts` when any pod restarts or CPU > 65% for 30s while p99 and error rate are flat.
Fix. Page on user-visible SLO violation or error budget burn; tune saturation alerts with duration thresholds and linkage to symptom metrics.
Logs only
Why it happens. Cross-service latency is additive; unstructured logs lack aggregation and causality without expensive correlation grep.
How it fails. Twelve-service checkout regression debug plan: “search Kibana for timeout” with no histograms or traces.
Fix. Metrics per hop (RED), distributed traces with propagated context, logs with `trace_id` for drill-down only where needed.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
Checkout SLO
Define an SLI, SLO, and error budget for `POST /checkout` (success rate and p99 latency). What metrics would you emit and what alert would page on-call vs land on a dashboard?
- about 7 min
Saturation before errors
p99 latency is fine but DB connection pool usage hovers at 95% for an hour. How do you alert and respond before users see widespread timeouts?
- about 8 min
Trace slow checkout
Checkout p99 jumped from 400ms to 2s across twelve microservices. Logs exist but grepping is slow. How would traces and metrics localize the regression?
- about 8 min
Release and error budget
Error budget for monthly API availability is 80% consumed on day 18. Product wants a large Friday deploy. How do error budgets guide the decision and what mitigations would you require?
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.