Messaging & Async

Scheduling & Workflows

Run delayed, recurring, or multi-step jobs reliably (cron, workflows, state machines).

Intermediate, about 55 minutes to read, 4 open questions to practise.

1Understand

The idea

Schedulers and workflows run work that must happen later, repeatedly, or across multiple steps without tying up an HTTP thread. Trials ending, invoice cron, approval chains, and refund pipelines need durable timers, retries, and idempotency under at-least-once execution.

A cron file on one laptop pod is not HA; await sleep(14 days) in Node dies on deploy. Interviews compare managed schedulers, delay queues, DB `run_at` polling, and workflow engines (Temporal, Step Functions) based on human-in-the-loop complexity.

Every step must tolerate duplicate delivery: cron overlap during failover, visibility timeout redelivery, or at-least-once queue semantics. Observability for missed crons is part of the design, silent failure until month-end is a classic production horror story.

Intuition

Job categories. One-shot delay: send reminder T+24h. Recurring: 0 2 * * * rollup. Workflow: state machine with human gates (AWAITING_MANAGER).

Timer storage. In-process timers lost on restart. Delay queue: SQS DelaySeconds / scheduled visibility; scheduler: Cloud Scheduler / k8s CronJob firing HTTP or enqueue; DB: scheduled_jobs(run_at, payload) polled by workers with FOR UPDATE SKIP LOCKED; workflow engine: durable timers in service history.

Idempotency. Keys like trial-end:{userId}:{period}; check status ALREADY_CHARGED before side effect; ledger upsert.

Visibility timeout. Worker dies mid-job → message visible again → second worker continues if steps checkpointed.

Workflow versioning. Changing step code mid-flight, engines replay from history; DIY cron needs explicit version on row.

Time zones. Bill on tenant TZ or UTC explicitly; DST jumps break naive local(2am), store UTC run_at.

Human steps. Timeouts escalate; compensate on reject (saga link).

Poison jobs. After max retries, DLQ with payload preserved; admin replay UI requeues with same idempotency key, workflows and queues share failure vocabulary.

Backpressure. If downstream billing API slows, delay queue depth rises, scale workers or throttle scheduler enqueue rate so timers do not pile unbounded in memory.

Human approvals. Manager step may wait days, workflow engine persists state; HTTP request already returned; user polls status URL.

Priority queues. Trial-end charge beats marketing email, separate queues so bulk campaigns never starve billing timers.

What you will be able to do

  • Pick cron vs queue delay vs workflow engine
  • Make job steps idempotent under retry
  • Observe lag and failed runs

When this comes up

  • Cron
  • Delayed job
  • Saga workflow
  • Exactly-once job

2See it

Architecture sketch

Durable workflow engine

Long-running steps with state, retries, and human tasks. Not a brittle cron chain.

100%
ClientServiceStoreQueue

Walkthrough

KYC multi-step

Survive process restarts

1 / 4

100%

Workflow starts and checkpoints “collect docs” in durable state.

Read the walkthrough as text

Worked example, SaaS trial ends day 14: charge or downgrade.

  1. Signup: insert trial_ends_at UTC; enqueue TrialEnd {userId} scheduled at that instant (Scheduler → SQS or Temporal timer).
  2. Never setTimeout in API.
  3. Worker fires: load user; if cancelled → exit 0; if TRIAL → call billing with idempotency key trial-end:{userId}.
  4. Duplicate fire: second run sees status=ACTIVE_PAID → no-op.
  5. Payment 503: exponential backoff; after N tries → DLQ + pager; user stays TRIAL with banner, honest UX.
  6. Observability: metric trial_end_jobs_processed; alert if no success in 24h window.
  7. Multi-step upgrade: optional workflow TRIAL → PAYMENT_PENDING → ACTIVE with webhook from Stripe as signal.

Nightly report variant: CronJob 02:00 UTC; idempotent report_runs row per date; alert if last_success stale > 26h. Refund workflow variant: three human steps with 72h SLA timer per step; escalate to finance queue on timeout.

What interviewers listen for: durable scheduler; idempotency; not in-request sleep; missed-run alerts; timezone clarity.

Napkin math

Workflow napkin

state machine with durable checkpoints

each step: enqueue → worker → record result

retry with backoff; max attempts then fail

human task = wait state, not busy poll

timers / SLA alerts on stuck runs

Long-running work lives in an engine, not a fragile cron chain.

3Answer it

How to answer in an interview

When delayed billing, cron, approvals, reminders:

  1. Job type, delay / recurring / multi-step.
  2. Infrastructure, scheduler, queue, workflow engine. Pick one with reason.
  3. Idempotency, concrete keys and status checks.
  4. Failure, retry, DLQ, human escalation.
  5. Observability, last success timestamp, lag.
  6. Clocks, UTC storage, tenant display rules.

Phrases: “durable timer,” “at-least-once safe handler,” “SKIP LOCKED job polling,” “alert on missed cron window.”

  1. Compare options, Scheduler vs Temporal in one sentence tied to human approval steps.
  1. Escalation. Name on-call owner when job stuck past SLA; finance cares about missed billing more than pod restarts.

Answer checklist

1. One-shot delay vs recurring vs multi-step workflow. 2. Trigger store (DB outbox, scheduler, workflow engine). 3. Idempotent steps and visibility timeout. 4. Failure / retry / human escalation. 5. Clock skew and daylight-save for schedules.

Tradeoffs & numbers

  • k8s CronJob: simple; missed if cluster unhealthy, external heartbeat monitor.
  • Delay queue: durable; max delay limits on some brokers; not always exact to the millisecond.
  • Temporal/Step Functions: best long workflows + humans; learning curve and cost.
  • DB polling: easy prototype; index run_at and cap batch size at scale.
  • Exactly-once jobs: rare; design at-least-once + idempotency.
  • When NOT workflow engine: single cron SQL report, Scheduler + idempotent script enough.
  • Clock skew: NTP matters; use external time source for billing cutoffs; never Date.now() alone across pods for legal deadlines.
  • Interview tip: contrast in-process timer vs queue timer with deploy-crash story, one sentence, high impact.
  • Runbook: name owner for missed cron and max acceptable staleness of derived tables, finance cares about this more than k8s.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Recurring cron

    Example. Aggregate usage metrics every night for billing; missed run loses revenue. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.

    How. UTC CronJob + idempotent insert per `billing_date`; PagerDuty if `last_success` > 26h stale. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.

  • Delayed single job

    Example. Abandoned cart email 2 hours after last activity. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.

    How. Schedule message with `deliverAt`; cancel if purchase event arrives first; idempotent send per cart id. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.

  • Multi-step workflow

    Example. Vendor payout: invoice submitted → finance approve → ACH sent. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.

    How. Persist `workflow_state`; each transition event handler idempotent; timeout escalates to manual queue. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.

  • At-least-once job runner

    Example. Worker killed mid-export after writing half the rows to S3. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.

    How. Checkpoint part numbers; redelivery resumes or overwrites idempotent S3 key `export/{id}/part-NN`. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

Cron spaghetti

Avoid. Five crons polling tables with no single state machine. Partial failures orphan rows.

Prefer. One durable workflow with explicit states, retries, and timeouts.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Sleep in request

    Why it happens. HTTP workers scale horizontally; in-memory sleep is lost on crash/deploy and holds precious connections.

    How it fails. Signup handler awaits seven days then charges card inside the same Lambda invocation.

    Fix. Persist schedule; delay queue or scheduler fires durable worker later.

  • Non-idempotent job

    Why it happens. Cron overlap and message redelivery double-charge, double-email, or duplicate ledger entries.

    How it fails. Billing job always POSTs charge without checking prior `invoice_id` or provider idempotency key.

    Fix. Status table, idempotency keys, upsert ledger rows safe on retry.

  • Silent cron failure

    Why it happens. Batch jobs fail quietly until downstream business processes discover missing data weeks later.

    How it fails. Nightly aggregation CronJob logs error to stdout only; no metric on last success.

    Fix. Heartbeat metric, alert on stale `last_success`, DLQ for failed job payloads.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons