Message Queues — Work Distribution and Spike Absorption

"Sign up" should return in 100ms. Sending the welcome email, generating the avatar, kicking off the analytics event, and warming the recommendations cache should not block that response. Enqueue and return; workers do the slow stuff. The user gets a fast page; you keep the work.

Black Friday traffic is 10× normal for two hours. Your DB and web tier can't auto-scale that fast. A queue is a buffer — requests pile up, workers drain at their normal rate, the system degrades into latency rather than collapsing into errors. You trade "instant" for "alive."

Worker died mid-job? Broker re-delivers. Email service was down? Worker raises an exception, doesn't ack, the message comes back. Combined with idempotent handlers (see below), you get free resilience.

Producer doesn't know how many workers exist or how long they take. Workers can be deployed and scaled independently. The team that owns the producer can ship without coordinating with the team that owns the worker.

Queues guarantee at-least-once delivery. Sometimes the worker finishes the job, then crashes before acking — the message comes back, and a second worker runs it again. Make every handler idempotent: store the job's unique ID and skip if already processed, or design the operation so running it twice is harmless (UPDATE order SET status='shipped' WHERE id=42 is naturally idempotent).

A message that fails repeatedly — bug in the handler, malformed payload, missing dependency — will block the queue forever if you keep retrying. After N failures, route it to a dead-letter queue for human inspection. Wire alerts on DLQ depth.

Set the timeout longer than your worst-case job duration plus a margin. Too short, and a slow job triggers a duplicate while it's still running. Too long, and a crashed worker holds messages hostage for ages. Many brokers let workers extend the timeout dynamically as work progresses.

If a job fails because a downstream service is down, retrying immediately doesn't help. Use exponential backoff between retries. After a retry budget, send to DLQ. If 1 message in 10,000 always fails — a malformed customer record from 2009 — let it land in DLQ and don't pretend it's transient.

It's tempting — SELECT FOR UPDATE SKIP LOCKED on a jobs table works, kind of. It's fine at low scale. At higher throughput it bottlenecks the DB, locks become contention hotspots, and you reinvent every queue feature poorly. Use a real broker once you're past "few jobs per second."

The exception: the outbox pattern, where an outbox table inside the same DB transaction as a state change feeds a real broker. That's a queue-as-a-table done right.