Messaging & Async

Stream & Batch Processing

Process continuous event streams or large batch jobs for analytics, ETL, and derived views.

Advanced, about 70 minutes to read, 4 open questions to practise.

1Understand

The idea

Stream and batch processing move product events into shapes humans and dashboards can use: aggregates, warehouse tables, search documents, fraud scores. The interview fork is freshness: if stakeholders need “yesterday’s revenue” by 6am, a partitioned batch job beats a Kafka cluster; if fraud must fire within seconds of a card swipe, you need consumers, windows, and partition-scaled parallelism.

Streams assume unbounded input, at-least-once delivery, and order only within a partition key. Batch assumes bounded input (a day’s parquet files), high throughput, and hours of latency tolerance. CDC (change data capture) bridges OLTP to the lake without polling every row.

Mature answers mention replay: when you ship a bug in window logic, you re-read retained log or archive and recompute idempotent sinks, pipelines without replay are one-way glue.

Intuition

Batch mental model. Input is finite: orders_date=2024-11-09 partition. Job reads, aggregates, writes warehouse table daily_revenue. Failure = rerun job idempotently (INSERT OVERWRITE or merge key). Schedule: cron/Airflow at 02:00 UTC. Cost driver: scan bytes, not millisecond latency.

Stream mental model. Topic is infinite log; partitions (userId, merchantId) give parallel consumers; ordering guaranteed only per partition. Consumer group: each partition → one active consumer instance at a time. State for aggregations lives in RocksDB / changelog topic, replay rebuilds state if you reset offsets.

Windows. Tumbling (non-overlapping 5m), sliding (overlapping), session (gap-based). Late events: watermarks and allowed lateness, say if you drop or correct.

CDC. Debezium-style capture from DB WAL → stream → sink to warehouse/search. Schema evolution (ADD COLUMN) needs compatibility plan.

Semantics. Producers: at-least-once common. Consumers: idempotent sink (UPSERT by (userId, windowStart)), or transactional Kafka + idempotent producer when justified. Global total order across all users = single partition bottleneck, reject unless prompt requires it.

Lambda / kappa (one line). Maintaining separate batch and speed layers doubles code paths, modern trend is “stream with replay as batch” when retention allows.

Backpressure & lag. Consumer lag metric is the heartbeat; scale consumers when lag > SLO; consider autoscaling on lag derivative, not CPU.

Schema evolution. Avro/Protobuf registry; consumers tolerate unknown fields; reprocess when compatibility breaks.

What you will be able to do

  • Choose batch vs stream from freshness needs
  • Design a consumer group / partition story
  • Plan replay when logic changes

When this comes up

  • Kafka consumer group
  • Windowed aggregation
  • ETL
  • Near-real-time pipeline

2See it

Architecture sketch

Log → consumers → sinks

Partitioned log retains history so you can replay after a bug.

100%
ServiceQueueStore

Walkthrough

Replay after a bug

Idempotent sinks make history useful

1 / 4

100%

Events append to a partitioned log. Ordering holds per partition key.

Read the walkthrough as text

Worked example, fraud alerts on payments + nightly finance report.

Events: PaymentAuthorized {userId, amount, ts, merchantId}; fraud rule: sum > $5k in rolling 5 minutes per user; finance needs daily totals by merchant.

  1. Split paths (interviewers love this): fraud = stream (SLA < 30s); finance = batch (SLA by 07:00).
  2. Stream setup: Kafka topic payments, partition key userId, 32 partitions; Flink/Streams consumer group fraud-rules.
  3. Window: sliding or tumbling 5m sum amount keyed by userId; state in embedded store; emit alert record when sum > threshold.
  4. Sink idempotency: alert id (userId, windowStart) upsert to alerts table, redelivery does not page twice.
  5. Batch path: nightly Spark/ BigQuery scheduled query on orders partition, no need to share Flink cluster with fraud.
  6. Bugfix story: deploy bad window boundary; replay from Kafka retention 7d or S3 archive; reset consumer group to T0; recompute alerts table with same idempotent keys.
  7. Ordering caveat: two events same user different partitions only if key wrong, always say “per-user order only.”

What interviewers listen for: explicit freshness SLAs; partition key choice; at-least-once + idempotent sink; batch for daily report; replay plan.

Napkin math

Stream napkin

log = Kafka (partitioned append)

partitions ≥ consumer parallelism

key = userId → order within partition

consumer lag = end_offset − committed

retain 7 days → replay after bad deploy

Offset + replay beat fragile in-memory pipelines.

3Answer it

How to answer in an interview

When analytics, pipelines, “real-time dashboard,” or CDC appears:

  1. Freshness SLA, seconds vs hours → stream vs batch first.
  2. Source / sink, Kafka, S3, warehouse, ES; name partition key.
  3. Ordering scope, per-key only; reject global order unless required.
  4. Delivery semantics, at-least-once; idempotent sink pattern.
  5. State / windows if aggregating, tumbling/sliding in one sentence.
  6. Replay / backfill, retention, reset offsets, overwrite derived table.
  7. Anti-overengineering, no Kafka for weekly CSV if batch suffices.

Phrases: “partition-scaled parallelism,” “watermark for lateness,” “CDC from OLTP,” “recomputable derived view.”

  1. Capacity, partitions ≈ max parallel consumers; picking 1 partition caps throughput.

Answer checklist

1. Latency need: batch hours vs stream seconds. 2. Source events and sink (warehouse, index, alerts). 3. Partitioning / ordering keys. 4. Exactly-once / idempotent sinks. 5. Replay and backfill.

Tradeoffs & numbers

  • Stream: low latency; ops (lag alerts, consumer rebalance, schema registry), worth when SLA < few minutes.
  • Batch: simpler/cheaper; stale until job completes, default for reporting.
  • Long Kafka retention (30d): replay-friendly; storage $$$, tier to S3 for archive.
  • Exactly-once Kafka transactions: stronger; latency and coupling, often idempotent sink enough.
  • Lambda architecture: dual code paths, maintenance tax.
  • When NOT to stream: executive PDF once daily; one-off migration; <1000 events/day.
  • Ordering bugs: mis-keyed partition sends related events to different partitions, debug with keyed trace samples.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Nightly batch ETL

    Example. CFO needs merchant revenue by 07:00 CET; sub-minute freshness is irrelevant. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.

    How. Partitioned warehouse job on `orders_dt`; idempotent overwrite; Airflow SLA alert if job misses window, no Kafka unless future real-time CFO dashboard appears.

  • Near-real-time aggregation

    Example. Ops wallboard: orders per minute per region during Black Friday. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.

    How. Kafka topic partitioned by `regionId`; tumbling 1m count; consumer scales with partitions; dashboard reads materialized view updated every few seconds.

  • CDC into warehouse

    Example. OLTP schema changes weekly; analytics must stay in sync without nightly full dump. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.

    How. Debezium on Postgres WAL → stream → BigQuery/Snowflake merge by primary key; schema registry for additive columns. Windowed aggregates per partition; checkpoint offsets; reprocess retained log on logic bugs.

  • Replay after bugfix

    Example. Fraud model v2 fixed double-counting; last 48h alerts wrong. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.

    How. Reset consumer to `T-48h` offsets; idempotent `upsert` alerts table; keep v1 topic archive on S3 for longer backfill if retention short.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

Firehose without replay

Avoid. Direct service→service calls with no log. Bug means data is gone.

Prefer. Durable log + idempotent consumers + ability to recompute.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Stream for a daily report

    Why it happens. Streaming stacks add brokers, consumer lag on-call, and schema ops when batch freshness already satisfies the business SLA.

    How it fails. Proposal: Kafka + Flink cluster to generate a single daily compliance PDF attached to email at 6am.

    Fix. Nightly partitioned batch ETL; add streaming only when a stated sub-minute or sub-hour SLA appears in requirements.

  • No replay

    Why it happens. Derived tables encode logic; without reprocessing history, a bad deploy permanently poisons metrics until manual SQL surgery.

    How it fails. Consumer commits offsets forward only; Kafka retention 24h; no archive, “we will fix forward.”

    Fix. Retain log or export to object store; idempotent sinks; documented reset-offset backfill runbook.

  • Assumes global order

    Why it happens. Parallel partitions interleave events; global ordering requires single partition and kills throughput.

    How it fails. “Payment events must be processed in exact global order across all merchants on 64 partitions.”

    Fix. Order per `accountId` partition key; design business rules for per-account sequencing only.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons