Data & Storage

Distributed Transactions & Sagas

Coordinate multi-step or multi-service updates without pretending 2PC is free.

Advanced, about 75 minutes to read, 4 open questions to practise.

1Understand

The idea

Distributed transactions in interviews are really invariant preservation when work spans machines: inventory must not go negative, money must not double-charge, and an order must not show PAID without a payment record. The default hammer, 2PC across microservices, is slow, blocks on coordinator failure, and is rarely the right first answer.

Prefer one database transaction when the data still fits one bounded context. When services are already split, sagas (forward steps + compensations) with visible PENDING states beat pretending everything is atomic. Transactional outbox fixes the crash between “committed order row” and “published event.”

“Exactly-once” on the wire is marketing; production means at-least-once delivery and idempotent consumers so effect happens once. Name intermediate states users may see during partial failure.

Intuition

Single-DB ACID. Order + line items + inventory decrement in one BEGIN…COMMIT, strongest, simplest. Split services later with clear pain, not day-one fashion.

Saga, orchestration vs choreography. Orchestration: central coordinator issues commands, tracks state machine PENDING → INVENTORY_RESERVED → PAID. Choreography: each service listens and emits events; less coupling, harder to debug stuck states. Both use compensations: ReleaseInventory, RefundPayment, compensations can fail; need runbooks.

2PC (mention, then avoid). Prepare phase locks resources; commit if all OK; coordinator dies → participants blocked. Used inside some databases; painful as app-level protocol across HTTP.

Transactional outbox. Same SQL txn: INSERT order + INSERT outbox(event). Relay worker polls or listens and publishes to broker, no lost events if API dies after commit.

Inbox pattern. Consumer stores message_id processed; skips duplicates.

TCC / try-confirm-cancel, interview one-liner: reserve → confirm or cancel; more complex than saga for many teams.

UX honesty: show Payment processing… until saga completes; do not flash success before payment ack.

Monitoring sagas. Dashboard counts per state (PENDING_PAYMENT, STUCK > 15m); alert on growth, ops intervenes before support inbox explodes.

Local vs global transactions. Even within microservices, each service uses single-DB txn per step; saga stitches them, do not nest 2PC inside each service call.

Reconciliation jobs. Nightly batch compares order totals vs payment ledger, sagas still need audit safety net for stuck edge cases.

Partial failure UX copy. Button says “Processing payment” not “Success” until saga terminal state, mobile offline retries must not assume HTTP 200 meant money moved.

What you will be able to do

  • Prefer single-store transactions when possible
  • Design a saga with compensations
  • Use outbox for reliable messaging after commit

When this comes up

  • Saga
  • Outbox
  • Compensating action
  • Exactly-once

2See it

Architecture sketch

Saga with outbox

Local commit + durable event beats 2PC across microservices for most interview designs.

100%
ClientServiceStoreQueue

Walkthrough

Order then pay

Compensate if payment fails

1 / 4

100%

Orders service commits order=PENDING and an outbox row in one local transaction.

Read the walkthrough as text

Worked example, checkout across Order, Payment, Inventory services.

Constraints: no oversell; no charge without order; user sees clear status.

  1. First question: can order+inventory live in one Postgres? If greenfield, yes, one txn, interview done for writes.
  2. Split path: Order service creates status=PENDING_PAYMENT, outbox OrderCreated in same txn.
  3. Orchestrator saga: step 1 reserve inventory (idempotent reserve:{orderId}); step 2 charge payment (idempotency-key: orderId); step 3 confirm order CONFIRMED; publish OrderConfirmed via outbox.
  4. Payment fails: compensate release inventory; order CANCELLED; user message honest.
  5. Duplicate messages: payment worker checks payment_status before charge; inventory reserve upsert.
  6. Crash story: order committed, outbox row exists, relay republishes, consumer idempotent.
  7. Reject: “We use 2PC HTTP across three services” without acknowledging blocking and coordinator SPOF.
  8. Observability: count orders per saga state; alert on PENDING > 10m backlog.

What interviewers listen for: single-DB first; saga steps named; compensations; outbox; idempotency; no exactly-once handwave.

Napkin math

Saga napkin

local tx in Orders DB + outbox row

publish OrderPlaced → Payments charges

fail charge → compensating CancelOrder

each step idempotent (dedupe by orderId)

avoid 2PC across services in interviews

Durable local commit + async steps beat distributed locks.

3Answer it

How to answer in an interview

When multi-service writes, checkout, or “all or nothing”:

  1. Can one DB txn cover it?, say yes/no with reason.
  2. If not, saga steps, states users see, compensations.
  3. Events, transactional outbox, not fire-and-forget.
  4. Idempotency keys per step and consumer.
  5. 2PC, acknowledge, then why you avoid as default.
  6. Stuck PENDING, monitoring, manual intervention, timeout to cancel.

Phrases: “at-least-once plus idempotent effect,” “compensating transaction,” “outbox in the same commit.”

  1. Stuck state, who pages when saga paused > 15m; ops dashboard by state count.

Answer checklist

1. What atomicity do you truly need? 2. Single-DB transaction vs saga across services. 3. Outbox / inbox for reliable event publish. 4. Compensations and user-visible intermediate states. 5. “Exactly-once” = at-least-once + idempotency.

Tradeoffs & numbers

  • Monolith DB txn: simplest consistency; team coupling until bounded context split.
  • Saga: team autonomy; eventual consistency; compensation failure edge cases.
  • Choreography: decoupled; opaque global state without orchestrator store.
  • Outbox: reliable publish; relay lag seconds; must dedupe consumers.
  • 2PC/TCC: stronger atomicity illusion; availability and latency cost, rare default.
  • When NOT saga: read-only aggregates; async notifications with no invariant.
  • Testing sagas: failure injection per step in staging, “payment always 503” validates compensations fire.
  • Interview tip: always draw the happy path *and* the halfway-failed path with user-visible state.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Single-DB ACID boundary

    Example. Early startup checkout: order lines and inventory decrement still in one monolith Postgres. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.

    How. One transaction commits both; defer saga until inventory service splits with measured need. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.

  • Choreographed saga

    Example. Inventory listens to OrderCreated; payment listens to InventoryReserved without central orchestrator. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.

    How. Each service local txn + event; compensating InventoryReleased on PaymentFailed; idempotent handlers. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.

  • Transactional outbox

    Example. API crashes after SQL commit but before Kafka publish, fulfillment never starts. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.

    How. Outbox row in same commit; relay publishes; consumer dedupes by `order_id` event id. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.

  • Idempotent consumer

    Example. Payment worker receives duplicate ChargeRequested after broker redelivery. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.

    How. Check `payments.status` for `order_id`; provider idempotency key matches; second delivery no-op. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

2PC across services

Avoid. Coordinator holds locks across Orders and Payments. Slow and brittle on failure.

Prefer. Local transactions + async saga/outbox with explicit compensation.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • 2PC everywhere

    Why it happens. Application-level 2PC blocks participants on coordinator or network faults and couples availability across all services.

    How it fails. Checkout coordinator runs prepare/commit HTTP across order, tax, payment, loyalty, email before responding.

    Fix. Colocate in one DB when possible; else orchestrated saga with compensations and idempotent steps.

  • Fire-and-forget event

    Why it happens. Process crash after DB commit but before publish creates orders that never ship, reconciled only by painful manual audits.

    How it fails. Insert order, return 201, then async goroutine publishes to Kafka without durability guarantee.

    Fix. Transactional outbox in same commit; relay worker publishes with retries.

  • Exactly-once myth

    Why it happens. Networks and consumers retry; claiming wire-level exactly-once avoids designing dedupe and idempotent effects.

    How it fails. “Kafka exactly-once means inventory service needs no idempotency.”.

    Fix. At-least-once delivery + idempotent consumer + business keys; clarify exactly-once applies to bounded contexts.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons