Security & Operations

Observability & SLOs

Know if the system is healthy: metrics, logs, traces, and error budgets.

Intermediate, about 55 minutes to read, 4 open questions to practise.

1Understand

The idea

Observability is how you know whether the product is healthy for real users, not whether a process is running. Dashboards full of CPU graphs without a user journey and an SLO are ops wallpaper. In system-design interviews, observability is the bridge between “we built retries and replicas” and “we can prove checkout stays under 500ms p99 when payment is slow.”

You instrument metrics (aggregates over time), logs (discrete events with context), and traces (causal paths across services). Those three pillars feed SLIs, measurable proxies like success rate or latency, and SLOs, numeric targets on those SLIs. The error budget (100% minus SLO) is how mature teams decide whether Friday’s deploy is allowed: budget left, ship with canaries; budget burned, fix before new risk.

Interviewers reward RED (rate, errors, duration) on APIs, USE (utilization, saturation, errors) on databases and queues, and alerts that page on symptoms (SLO violation, payment error spike) not causes (one pod restarted). Tie every probe to a journey you already named in capacity planning, “add tracing on checkout” lands harder after you sized checkout QPS and latency budget.

Intuition

Start from the journey, not the toolkit. Pick one critical path, login, checkout, feed load, upload complete, and ask what “bad” means in user language: timeouts, wrong totals, stale UI, partial failure. That becomes your SLI list. Avoid measuring everything; measure what gates revenue or trust.

SLI → SLO → budget. An SLI is a measurement: successful_checkouts / total_checkout_attempts over a rolling window, or histogram http_request_duration_seconds for route=checkout. An SLO is a target: “99.95% checkout success per 30 days” and “p99 latency < 2s.” Error budget is the allowed un-good fraction, for 99.95% monthly availability, roughly 22 minutes of bad minutes-equivalent per month (order-of-magnitude; say you know it is approximate). Releases and experiments spend budget; postmortems explain spend.

Metrics (when to use). Counters and histograms for rates and percentiles; low cost at aggregate cardinality (route, status, not unbounded user_id on every series). RED per service: request rate, error ratio, duration percentiles. USE on resources: DB connection pool utilization, disk queue depth, Kafka consumer lag, saturation often predicts user errors before error rate jumps.

Logs. Structured JSON with trace_id, request_id, tenant_id (hashed if PII), level, stable field names. Logs answer “what happened for this id” after metrics/traces point at a spike. High-cardinality dimensions in log indexes (raw email per line) explode cost, aggregate in metrics, detail in sampled logs.

Traces. OpenTelemetry / traceparent propagation across gateway → app → DB → payment client. One slow checkout trace showing 1.8s in inventory retries beats grep across twelve unstructured log files. Sample 1-5% happy paths; always retain errors or tail p99.

Alerting philosophy. Page humans for user-visible SLO burn or multiplicative error rate; ticket for trends. “CPU > 70% for 1m” without checkout impact is noise. Multi-window burn rates (Google SRE book) catch fast and slow budget leaks. Runbooks link from alerts to dashboards and ownership.

What you will be able to do

  • Define an SLI/SLO for a user journey
  • Instrument latency, errors, and saturation
  • Use traces for multi-service requests

When this comes up

  • RED / USE metrics
  • p99 latency
  • Distributed tracing
  • Error budget

2See it

Architecture sketch

Three pillars into alerts

Instrument the journey you sized in capacity planning.

100%
ServiceOtherGatewayClient

Walkthrough

Checkout burns budget

Symptom alert beats cause spam

1 / 4

100%

Checkout API emits rate, errors, duration (RED). Saturation shows on DB/queue (USE).

Read the walkthrough as text

Worked example, checkout SLO for a payments-integrated storefront.

Product: POST /checkout authorizes card, reserves inventory, sends confirmation email async. Interview asks: “How do you know checkout is healthy?”

  1. Name the journey SLIs (say assumptions): success = HTTP 2xx and payment_status=authorized; latency = server-side time to response; exclude client aborts if policy says so.
  2. SLO targets: “99.95% successful checkouts per rolling 30d; p99 latency < 2s for the sync path (email async).” Connect to business: missed SLO = lost sales + chargeback risk.
  3. Instrumentation: API middleware histogram checkout_duration_seconds with buckets to 5s; counter checkout_outcomes{status=success|payment_decline|inventory_conflict|5xx}; propagate trace_id into payment HTTP client and inventory gRPC.
  4. Saturation (USE): alert DB pool > 85% for 10m; payment worker queue lag > 30s; before user-visible 503s, interviewers like “leading indicators.”
  5. Error budget: dashboard shows budget remaining; deploy policy, canary 5% traffic; auto-halt if 5xx rate 2× baseline for 15m or fast burn alert fires.
  6. Incident: p99 jumps from 400ms → 1.2s; on-call opens trace sample, sees inventory client retry storm on 429; fix backoff, not “scale API pods.”
  7. Logs: correlate single failed order_id via trace_id across services for support tickets.

What interviewers listen for: numeric SLO, not “Grafana”; symptom-based paging; metrics + traces + structured logs; explicit async boundary (email off critical SLI); you distinguish availability SLI from latency SLI.

Napkin math

SLO napkin

SLI = successful checkouts / total checkouts

SLO = 99.9% over 30 days

error budget ≈ 0.1% ≈ 43 min downtime/month

alert on burn rate, not single pod restart

Metrics + logs + traces feed SLIs. Page on user symptoms.

3Answer it

How to answer in an interview

When prompts mention monitoring, on-call, reliability, or “how do you debug production”:

  1. Pick one user journey, checkout, signup, search, upload, match the product story.
  2. Define 1-2 SLIs, success rate, latency percentile, optionally freshness.
  3. State SLO numbers, monthly or weekly window; mention error budget in one sentence.
  4. Three pillars, which metric names/histograms, which log fields, trace propagation on critical hops.
  5. Saturation, pool, lag, disk, alert before pure error spikes.
  6. Alert + release policy, page on budget burn / user errors; canary tied to SLO.
  7. Debug story, “trace shows payment retry” beats “we would grep logs.”

Phrases to use: “symptom-based alerting,” “error budget,” “RED on the API, USE on the database,” “propagate traceparent,” “high-cardinality is expensive, aggregate in metrics.”

Answer checklist

1. User-facing SLI (availability, latency). 2. SLO and error budget. 3. Metrics on the critical path. 4. Logs vs traces for debugging. 5. Alerts that page humans vs dashboards.

Tradeoffs & numbers

  • Tight SLO (99.99%): fewer outages tolerated; slower feature velocity; more redundancy cost, justify with revenue tier.
  • Loose SLO (99%): cheap ops; users feel pain, OK only if product is internal or non-critical.
  • High-cardinality labels (user_id on every metric series): precise debugging, unsustainable cost, prefer exemplars or sampled traces.
  • 100% trace sampling: great forensics, 10-30% overhead, sample errors always; head-based 1-5% on success.
  • Log everything: compliance vs dollars; retention tiers (7d hot, 90d cold).
  • Synthetic probes: catch regional DNS/LB failures; miss real user device mix, combine with RUM if asked.
  • When NOT to over-instrument: early MVP with 100 DAU, one golden-signal dashboard + basic alerts; skip full tracing mesh until scale demands it.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • API latency SLO

    Example. Mobile home feed p99 must stay under 250ms monthly; spikes after deploy blocked the last release. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.

    How. Histogram on `feed_handler_duration_seconds`; SLO on p99; canary compares canary vs baseline p99; trace slow spans to DB vs cache miss.

  • Saturation / capacity alerts

    Example. Checkout errors rose only after connection pool exhausted for six minutes, CPU was idle. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.

    How. USE metrics: pool active/limit, queue depth, Kafka lag; alert at 80% sustained; runbook scale replicas or fix connection leaks before user 503s.

  • Trace a slow request

    Example. Support ticket: one user, checkout took 12s; logs alone show timeouts in three services. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.

    How. Find `trace_id` from gateway access log; waterfall shows payment SDK retry loop; fix idempotent retries with cap, not add Redis “for speed.”

  • Error budget for releases

    Example. Team wants Friday deploy; budget is 85% consumed after bad canary Tuesday. Tie RED/USE metrics to a user journey SLO; alert on symptoms (error budget burn) not only pod restarts.

    How. Show burn rate math; freeze feature flags; ship only fix-forward; postmortem action items before next risky change. SLI → SLO → error budget; exemplar trace on slow checkout; page when budget burns, not on CPU noise.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

What you page on

Avoid. Page on every pod restart and CPU blip. Noise trains people to ignore pages.

Prefer. Page on user-journey SLOs and payment error spikes. Debug with traces.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • No SLO

    Why it happens. Dashboards without targets cannot rank incidents, justify headcount, or gate releases, on-call debates vibes at 3am.

    How it fails. “We have Prometheus and Grafana on 400 panels” but no stated availability or latency goal for signup or checkout.

    Fix. Pick one journey SLI; set SLO (e.g. 99.9% / 30d); track error budget; page on multi-window burn, not raw CPU.

  • Alert noise

    Why it happens. Pager fatigue trains humans to ignore pages; real SLO breaches get lost in infra chatter. Metrics without SLO/error budget do not drive release decisions.

    How it fails. Pages `#prod-alerts` when any pod restarts or CPU > 65% for 30s while p99 and error rate are flat.

    Fix. Page on user-visible SLO violation or error budget burn; tune saturation alerts with duration thresholds and linkage to symptom metrics.

  • Logs only

    Why it happens. Cross-service latency is additive; unstructured logs lack aggregation and causality without expensive correlation grep.

    How it fails. Twelve-service checkout regression debug plan: “search Kibana for timeout” with no histograms or traces.

    Fix. Metrics per hop (RED), distributed traces with propagated context, logs with `trace_id` for drill-down only where needed.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons