1Understand
The idea
Distributed transactions in interviews are really invariant preservation when work spans machines: inventory must not go negative, money must not double-charge, and an order must not show PAID without a payment record. The default hammer, 2PC across microservices, is slow, blocks on coordinator failure, and is rarely the right first answer.
Prefer one database transaction when the data still fits one bounded context. When services are already split, sagas (forward steps + compensations) with visible PENDING states beat pretending everything is atomic. Transactional outbox fixes the crash between “committed order row” and “published event.”
“Exactly-once” on the wire is marketing; production means at-least-once delivery and idempotent consumers so effect happens once. Name intermediate states users may see during partial failure.
Intuition
Single-DB ACID. Order + line items + inventory decrement in one BEGIN…COMMIT, strongest, simplest. Split services later with clear pain, not day-one fashion.
Saga, orchestration vs choreography. Orchestration: central coordinator issues commands, tracks state machine PENDING → INVENTORY_RESERVED → PAID. Choreography: each service listens and emits events; less coupling, harder to debug stuck states. Both use compensations: ReleaseInventory, RefundPayment, compensations can fail; need runbooks.
2PC (mention, then avoid). Prepare phase locks resources; commit if all OK; coordinator dies → participants blocked. Used inside some databases; painful as app-level protocol across HTTP.
Transactional outbox. Same SQL txn: INSERT order + INSERT outbox(event). Relay worker polls or listens and publishes to broker, no lost events if API dies after commit.
Inbox pattern. Consumer stores message_id processed; skips duplicates.
TCC / try-confirm-cancel, interview one-liner: reserve → confirm or cancel; more complex than saga for many teams.
UX honesty: show Payment processing… until saga completes; do not flash success before payment ack.
Monitoring sagas. Dashboard counts per state (PENDING_PAYMENT, STUCK > 15m); alert on growth, ops intervenes before support inbox explodes.
Local vs global transactions. Even within microservices, each service uses single-DB txn per step; saga stitches them, do not nest 2PC inside each service call.
Reconciliation jobs. Nightly batch compares order totals vs payment ledger, sagas still need audit safety net for stuck edge cases.
Partial failure UX copy. Button says “Processing payment” not “Success” until saga terminal state, mobile offline retries must not assume HTTP 200 meant money moved.
What you will be able to do
- Prefer single-store transactions when possible
- Design a saga with compensations
- Use outbox for reliable messaging after commit
When this comes up
- Saga
- Outbox
- Compensating action
- Exactly-once
2See it
Architecture sketch
Saga with outbox
Local commit + durable event beats 2PC across microservices for most interview designs.
Walkthrough
Order then pay
Compensate if payment fails
1 / 4
Orders service commits order=PENDING and an outbox row in one local transaction.
Read the walkthrough as text
Worked example, checkout across Order, Payment, Inventory services.
Constraints: no oversell; no charge without order; user sees clear status.
- First question: can order+inventory live in one Postgres? If greenfield, yes, one txn, interview done for writes.
- Split path: Order service creates
status=PENDING_PAYMENT, outboxOrderCreatedin same txn. - Orchestrator saga: step 1 reserve inventory (idempotent
reserve:{orderId}); step 2 charge payment (idempotency-key: orderId); step 3 confirm orderCONFIRMED; publishOrderConfirmedvia outbox. - Payment fails: compensate release inventory; order
CANCELLED; user message honest. - Duplicate messages: payment worker checks
payment_statusbefore charge; inventory reserve upsert. - Crash story: order committed, outbox row exists, relay republishes, consumer idempotent.
- Reject: “We use 2PC HTTP across three services” without acknowledging blocking and coordinator SPOF.
- Observability: count orders per saga state; alert on
PENDING > 10mbacklog.
What interviewers listen for: single-DB first; saga steps named; compensations; outbox; idempotency; no exactly-once handwave.
Napkin math
local tx in Orders DB + outbox row
publish OrderPlaced → Payments charges
fail charge → compensating CancelOrder
each step idempotent (dedupe by orderId)
avoid 2PC across services in interviews
Durable local commit + async steps beat distributed locks.
3Answer it
How to answer in an interview
When multi-service writes, checkout, or “all or nothing”:
- Can one DB txn cover it?, say yes/no with reason.
- If not, saga steps, states users see, compensations.
- Events, transactional outbox, not fire-and-forget.
- Idempotency keys per step and consumer.
- 2PC, acknowledge, then why you avoid as default.
- Stuck PENDING, monitoring, manual intervention, timeout to cancel.
Phrases: “at-least-once plus idempotent effect,” “compensating transaction,” “outbox in the same commit.”
- Stuck state, who pages when saga paused > 15m; ops dashboard by state count.
Answer checklist
Tradeoffs & numbers
- Monolith DB txn: simplest consistency; team coupling until bounded context split.
- Saga: team autonomy; eventual consistency; compensation failure edge cases.
- Choreography: decoupled; opaque global state without orchestrator store.
- Outbox: reliable publish; relay lag seconds; must dedupe consumers.
- 2PC/TCC: stronger atomicity illusion; availability and latency cost, rare default.
- When NOT saga: read-only aggregates; async notifications with no invariant.
- Testing sagas: failure injection per step in staging, “payment always 503” validates compensations fire.
- Interview tip: always draw the happy path *and* the halfway-failed path with user-visible state.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Single-DB ACID boundary
Example. Early startup checkout: order lines and inventory decrement still in one monolith Postgres. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.
How. One transaction commits both; defer saga until inventory service splits with measured need. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.
Choreographed saga
Example. Inventory listens to OrderCreated; payment listens to InventoryReserved without central orchestrator. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.
How. Each service local txn + event; compensating InventoryReleased on PaymentFailed; idempotent handlers. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.
Transactional outbox
Example. API crashes after SQL commit but before Kafka publish, fulfillment never starts. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.
How. Outbox row in same commit; relay publishes; consumer dedupes by `order_id` event id. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.
Idempotent consumer
Example. Payment worker receives duplicate ChargeRequested after broker redelivery. Prefer one local transaction when possible; otherwise saga/outbox with visible `PENDING` states and idempotent consumers.
How. Check `payments.status` for `order_id`; provider idempotency key matches; second delivery no-op. Saga steps with compensations; outbox for publish; users may see intermediate states. Say them out loud.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
2PC across services
Avoid. Coordinator holds locks across Orders and Payments. Slow and brittle on failure.
Prefer. Local transactions + async saga/outbox with explicit compensation.
Weak sketch
Bad
Stronger sketch
Good
2PC everywhere
Why it happens. Application-level 2PC blocks participants on coordinator or network faults and couples availability across all services.
How it fails. Checkout coordinator runs prepare/commit HTTP across order, tax, payment, loyalty, email before responding.
Fix. Colocate in one DB when possible; else orchestrated saga with compensations and idempotent steps.
Fire-and-forget event
Why it happens. Process crash after DB commit but before publish creates orders that never ship, reconciled only by painful manual audits.
How it fails. Insert order, return 201, then async goroutine publishes to Kafka without durability guarantee.
Fix. Transactional outbox in same commit; relay worker publishes with retries.
Exactly-once myth
Why it happens. Networks and consumers retry; claiming wire-level exactly-once avoids designing dedupe and idempotent effects.
How it fails. “Kafka exactly-once means inventory service needs no idempotency.”.
Fix. At-least-once delivery + idempotent consumer + business keys; clarify exactly-once applies to bounded contexts.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
One database txn
Checkout creates an order row, line items, and decrements inventory, all in one Postgres today. A teammate wants microservices immediately with 2PC. Argue for the single-DB boundary now and when you would split later.
- about 10 min
Checkout saga
Order, payment, and inventory are three separate services. Design a saga for happy path and payment failure: steps, compensations, and what status the user sees while pending.
- about 8 min
Transactional outbox
After inserting an order in SQL, you must publish `OrderCreated` to Kafka for fulfillment. The API sometimes crashes after commit but before publish. Design the outbox pattern and relay.
- about 8 min
Exactly-once effect
A payment consumer may receive duplicate `ChargeRequested` messages. The partner claims Kafka “exactly-once” solves everything. Explain what you still implement in the consumer and why.
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.