Data & Storage

Media Uploads & Delivery

Accept large files safely (direct-to-object-store, resumable upload) and serve them via CDN.

Intermediate, about 60 minutes to read, 4 open questions to practise.

1Understand

The idea

Media uploads at scale send bytes to object storage (S3/GCS) while the API tier only authorizes, records metadata, and queues processing. Streaming gigabytes through stateless pods wastes bandwidth, blows timeouts, and bypasses multipart/resume features clients need on mobile.

Presigned URLs scope bucket, key prefix, size, and content-type for a short TTL. Multipart uploads retry parts independently. SQL holds video_id, owner, status, s3_key; bytes live in the bucket. Transcode, virus scan, and thumbnails run async after complete, response returns processing, not HLS ready.

Public renditions often sit on CDN; private originals need authz-checked short-lived GET URLs.

Think in three phases: authorize upload, land bytes durably, process asynchronously. Skipping phase separation is how startups melt their API fleet during a viral video contest.

Intuition

Direct upload flow. Client requests session → API verifies quota/authz → mints presigned PUT/multipart URLs → client uploads parallel to S3 → client calls complete with ETags → API finalizes object, enqueues pipeline.

Why not proxy. client_max_body_size 2G on nginx ties connections minutes; autoscaling on upload bytes is wrong metric; API memory risk.

Multipart. Parts 5-100MB; retry failed part only; CompleteMultipartUpload assembles; list parts for resume after app kill.

Metadata states. INIT → UPLOADED → PROCESSING → READY | FAILED; UI polls or webhook; never block HTTP on ffmpeg.

Post-upload workers. Transcode HLS renditions to s3://bucket/{id}/hls/; update SQL; CDN cache public derivatives; virus scan quarantine before READY.

Authz download. Check owner/coach role → presigned GET 5m or CloudFront signed URL; no permanent public URL for medical/legal content.

Abuse: max size, MIME validation, per-user daily quota, malware scan.

Content integrity. Record SHA-256 on complete; compare after transcode; detect tampered client metadata claiming wrong MIME.

Cost control. Egress from S3 to transcode workers in same region; lifecycle policy moves cold originals to IA/Glacier per retention SKU.

Client responsibility. Mobile uploads parts, server never sees bytes; client must handle retry UX; server validates complete callback against S3 head object size.

Thumbnail path. Cheap image derivatives may sync-generate; full video transcode stays async, tier processing by asset type.

What you will be able to do

  • Use direct-to-object-store uploads with short-lived credentials
  • Separate metadata DB from blob storage
  • Process media asynchronously after upload

When this comes up

  • Presigned URL
  • Multipart upload
  • Transcoding
  • Thumbnails

2See it

Architecture sketch

Direct upload + async process

Browser puts bytes to object storage; workers transcode; CDN serves.

100%
ClientServiceStoreQueueGateway

Walkthrough

Upload a video

Presign → PUT → process

1 / 4

100%

Client asks API for a presigned URL. API records metadata (owner, status=pending).

Read the walkthrough as text

Worked example, fitness app: 500MB workout video from mobile.

  1. POST /videos/upload-session {sizeBytes, contentType}, authz + plan quota.
  2. API inserts status=INIT, generates videoId, returns multipart presigned URLs for 32 parts.
  3. Client uploads parts directly to S3 in parallel; retries part 17 only on LTE blip.
  4. POST /videos/{id}/complete with part ETags, S3 complete; SQL UPLOADED; publish VideoUploaded to queue.
  5. Worker transcode → HLS keys; SQL READY; push notification optional async.
  6. GET /videos/{id} metadata + CDN URL for HLS playlist when public; coach-only uses signed GET after authz.
  7. Failure: transcode fail → FAILED + support id; object retained for retry job.
  8. Abuse: reject oversize session; MIME sniff; per-user daily upload cap enforced at session mint.
  9. CDN: public HLS on CDN; original raw on private bucket only.
  10. Legal hold: some objects skip lifecycle delete, metadata flag blocks purge jobs.

What interviewers listen for: presigned direct; multipart; metadata/bytes split; async transcode; CDN vs private signed reads.

Napkin math

Upload sizing

max object = 100 MB video

direct-to-S3 presigned PUT

API stores metadata only

async transcode via queue

CDN for playback URLs

Never stream large blobs through your app tier.

3Answer it

How to answer in an interview

When video, images, large files, user-generated content:

  1. Upload path, presigned direct vs proxy and why.
  2. Authz & key layout, tenant/user/uuid prefix.
  3. Resumable multipart for large/flaky networks.
  4. DB metadata vs object bytes.
  5. Async pipeline, queue, statuses, UX while processing.
  6. Delivery, CDN public derivatives vs signed private.
  7. Abuse limits, size, MIME, scan.

Phrases: “API never proxies gigabytes,” “complete callback then queue,” “short-lived presigned GET.”

  1. Progress UX: expose upload percent from client part completion; transcode percent from worker events.

Answer checklist

1. Client uploads where (direct to blob vs via API)? 2. Authz for upload and download URLs. 3. Processing pipeline (virus scan, transcode) via queue. 4. Metadata in DB; bytes in object store. 5. Delivery through CDN; failed processing UX.

Tradeoffs & numbers

  • Presigned: offloads bandwidth; leak risk if TTL too long, scope prefix + short expiry.
  • Multipart: resilient; client complexity.
  • Sync transcode: simple demo; production timeouts and angry users.
  • Scan before publish: security vs time-to-ready, quarantine bucket pattern.
  • CDN on private: signed URLs; mis-cache leaks, careful cache key policy.
  • When NOT multipart: <5MB avatar, single PUT presign sufficient.
  • Edge malware: scanning every upload in sync adds minutes, async quarantine bucket is the usual compromise.
  • Interview tip: separate upload-complete from transcode-ready in the API response schema, two timestamps.
  • Cost model: egress and transcode minutes dominate COGS, per-GB caps belong in product tiering story.
  • Duplicate complete callbacks: treat complete POST as idempotent on videoId, mobile will retry.
  • Webhook optional: media.ready event beats polling when transcode completes.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Presigned / direct upload

    Example. Users upload 15MB profile banners; API pods are 512MB memory limit. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.

    How. Presigned PUT to `uploads/{userId}/{uuid}`; API only validates MIME/size policy before signing. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.

  • Resumable multipart

    Example. Documentary filmmakers upload 40GB source on hotel Wi-Fi with drops every 20 minutes. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.

    How. Multipart with list-parts resume; client tracks completed part numbers; complete when all ETags collected. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.

  • Transcode pipeline

    Example. User uploads portrait video; product requires landscape HLS for TV app. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.

    How. Queue after complete; worker ffmpeg → multiple bitrates; SQL status; CDN serves `master.m3u8` when READY. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.

  • Authenticated download

    Example. Legal discovery videos visible only to case members, not public CDN. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.

    How. Authz check case membership; presigned GET 300s; audit log; object bucket policy denies anonymous. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

Proxy vs direct

Avoid. Multipart through Node/Rails to disk then S3. Memory and timeouts explode.

Prefer. Presigned direct upload; app stores keys and kicks async jobs.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Proxy huge uploads

    Why it happens. Proxying large uploads consumes app CPU, bandwidth, and connection slots; autoscaling on upload spikes is expensive and fragile.

    How it fails. Express route buffers entire 3GB upload in memory then streams to S3 on single t3.medium.

    Fix. Presigned direct upload; API authorizes and records metadata only.

  • Blobs in the DB

    Why it happens. OLTP databases replicate and backup whole rows; gigabyte blobs bloat WAL, slow failover, and explode backup cost.

    How it fails. Store raw MP4 in Postgres BYTEA column indexed by primary key.

    Fix. Object store for bytes; SQL stores pointer, checksum, status, and ownership metadata.

  • Sync transcode

    Why it happens. Transcoding can exceed HTTP timeout and leaves user staring at spinner; failures retry whole upload.

    How it fails. Upload endpoint returns 200 only after ffmpeg produces HLS inside the request handler.

    Fix. Acknowledge upload complete quickly; async worker transcode; client polls status or receives push.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons