1Understand
The idea
Media uploads at scale send bytes to object storage (S3/GCS) while the API tier only authorizes, records metadata, and queues processing. Streaming gigabytes through stateless pods wastes bandwidth, blows timeouts, and bypasses multipart/resume features clients need on mobile.
Presigned URLs scope bucket, key prefix, size, and content-type for a short TTL. Multipart uploads retry parts independently. SQL holds video_id, owner, status, s3_key; bytes live in the bucket. Transcode, virus scan, and thumbnails run async after complete, response returns processing, not HLS ready.
Public renditions often sit on CDN; private originals need authz-checked short-lived GET URLs.
Think in three phases: authorize upload, land bytes durably, process asynchronously. Skipping phase separation is how startups melt their API fleet during a viral video contest.
Intuition
Direct upload flow. Client requests session → API verifies quota/authz → mints presigned PUT/multipart URLs → client uploads parallel to S3 → client calls complete with ETags → API finalizes object, enqueues pipeline.
Why not proxy. client_max_body_size 2G on nginx ties connections minutes; autoscaling on upload bytes is wrong metric; API memory risk.
Multipart. Parts 5-100MB; retry failed part only; CompleteMultipartUpload assembles; list parts for resume after app kill.
Metadata states. INIT → UPLOADED → PROCESSING → READY | FAILED; UI polls or webhook; never block HTTP on ffmpeg.
Post-upload workers. Transcode HLS renditions to s3://bucket/{id}/hls/; update SQL; CDN cache public derivatives; virus scan quarantine before READY.
Authz download. Check owner/coach role → presigned GET 5m or CloudFront signed URL; no permanent public URL for medical/legal content.
Abuse: max size, MIME validation, per-user daily quota, malware scan.
Content integrity. Record SHA-256 on complete; compare after transcode; detect tampered client metadata claiming wrong MIME.
Cost control. Egress from S3 to transcode workers in same region; lifecycle policy moves cold originals to IA/Glacier per retention SKU.
Client responsibility. Mobile uploads parts, server never sees bytes; client must handle retry UX; server validates complete callback against S3 head object size.
Thumbnail path. Cheap image derivatives may sync-generate; full video transcode stays async, tier processing by asset type.
What you will be able to do
- Use direct-to-object-store uploads with short-lived credentials
- Separate metadata DB from blob storage
- Process media asynchronously after upload
When this comes up
- Presigned URL
- Multipart upload
- Transcoding
- Thumbnails
2See it
Architecture sketch
Direct upload + async process
Browser puts bytes to object storage; workers transcode; CDN serves.
Walkthrough
Upload a video
Presign → PUT → process
1 / 4
Client asks API for a presigned URL. API records metadata (owner, status=pending).
Read the walkthrough as text
Worked example, fitness app: 500MB workout video from mobile.
- POST /videos/upload-session
{sizeBytes, contentType}, authz + plan quota. - API inserts
status=INIT, generatesvideoId, returns multipart presigned URLs for 32 parts. - Client uploads parts directly to S3 in parallel; retries part 17 only on LTE blip.
- POST /videos/{id}/complete with part ETags, S3 complete; SQL
UPLOADED; publishVideoUploadedto queue. - Worker transcode → HLS keys; SQL
READY; push notification optional async. - GET /videos/{id} metadata + CDN URL for HLS playlist when public; coach-only uses signed GET after authz.
- Failure: transcode fail →
FAILED+ support id; object retained for retry job. - Abuse: reject oversize session; MIME sniff; per-user daily upload cap enforced at session mint.
- CDN: public HLS on CDN; original raw on private bucket only.
- Legal hold: some objects skip lifecycle delete, metadata flag blocks purge jobs.
What interviewers listen for: presigned direct; multipart; metadata/bytes split; async transcode; CDN vs private signed reads.
Napkin math
max object = 100 MB video
direct-to-S3 presigned PUT
API stores metadata only
async transcode via queue
CDN for playback URLs
Never stream large blobs through your app tier.
3Answer it
How to answer in an interview
When video, images, large files, user-generated content:
- Upload path, presigned direct vs proxy and why.
- Authz & key layout,
tenant/user/uuidprefix. - Resumable multipart for large/flaky networks.
- DB metadata vs object bytes.
- Async pipeline, queue, statuses, UX while processing.
- Delivery, CDN public derivatives vs signed private.
- Abuse limits, size, MIME, scan.
Phrases: “API never proxies gigabytes,” “complete callback then queue,” “short-lived presigned GET.”
- Progress UX: expose upload percent from client part completion; transcode percent from worker events.
Answer checklist
Tradeoffs & numbers
- Presigned: offloads bandwidth; leak risk if TTL too long, scope prefix + short expiry.
- Multipart: resilient; client complexity.
- Sync transcode: simple demo; production timeouts and angry users.
- Scan before publish: security vs time-to-ready, quarantine bucket pattern.
- CDN on private: signed URLs; mis-cache leaks, careful cache key policy.
- When NOT multipart: <5MB avatar, single PUT presign sufficient.
- Edge malware: scanning every upload in sync adds minutes, async quarantine bucket is the usual compromise.
- Interview tip: separate upload-complete from transcode-ready in the API response schema, two timestamps.
- Cost model: egress and transcode minutes dominate COGS, per-GB caps belong in product tiering story.
- Duplicate complete callbacks: treat
completePOST as idempotent onvideoId, mobile will retry. - Webhook optional:
media.readyevent beats polling when transcode completes.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Presigned / direct upload
Example. Users upload 15MB profile banners; API pods are 512MB memory limit. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.
How. Presigned PUT to `uploads/{userId}/{uuid}`; API only validates MIME/size policy before signing. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.
Resumable multipart
Example. Documentary filmmakers upload 40GB source on hotel Wi-Fi with drops every 20 minutes. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.
How. Multipart with list-parts resume; client tracks completed part numbers; complete when all ETags collected. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.
Transcode pipeline
Example. User uploads portrait video; product requires landscape HLS for TV app. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.
How. Queue after complete; worker ffmpeg → multiple bitrates; SQL status; CDN serves `master.m3u8` when READY. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.
Authenticated download
Example. Legal discovery videos visible only to case members, not public CDN. Presigned multipart to object storage; metadata in SQL; transcode/thumbnail async with status polling, not bytes through API pods.
How. Authz check case membership; presigned GET 300s; audit log; object bucket policy denies anonymous. Client PUTs to S3; API records `processing`; virus scan/transcode workers; CDN only for public renditions.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
Proxy vs direct
Avoid. Multipart through Node/Rails to disk then S3. Memory and timeouts explode.
Prefer. Presigned direct upload; app stores keys and kicks async jobs.
Weak sketch
Bad
Stronger sketch
Good
Proxy huge uploads
Why it happens. Proxying large uploads consumes app CPU, bandwidth, and connection slots; autoscaling on upload spikes is expensive and fragile.
How it fails. Express route buffers entire 3GB upload in memory then streams to S3 on single t3.medium.
Fix. Presigned direct upload; API authorizes and records metadata only.
Blobs in the DB
Why it happens. OLTP databases replicate and backup whole rows; gigabyte blobs bloat WAL, slow failover, and explode backup cost.
How it fails. Store raw MP4 in Postgres BYTEA column indexed by primary key.
Fix. Object store for bytes; SQL stores pointer, checksum, status, and ownership metadata.
Sync transcode
Why it happens. Transcoding can exceed HTTP timeout and leaves user staring at spinner; failures retry whole upload.
How it fails. Upload endpoint returns 200 only after ffmpeg produces HLS inside the request handler.
Fix. Acknowledge upload complete quickly; async worker transcode; client polls status or receives push.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
Presigned direct upload
Users upload profile images up to 10MB. Your API tier is stateless and should not proxy file bytes. Describe the client-API-object-store flow including authz and completion callback.
- about 8 min
Resumable multipart
Mobile clients upload 800MB workout videos on flaky LTE. How do multipart uploads, part retries, and completion work with presigned URLs?
- about 8 min
Transcode after upload
After raw video lands in object storage, you must produce HLS renditions and thumbnails. What should the upload API return, and how does the user know when the video is playable?
- about 7 min
Private media download
Medical images must only be viewable by the patient and assigned clinicians. Public CDN caching is inappropriate for originals. How do authorized clients fetch bytes?
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.