Reliability patterns for queue-based job processing?

WHAT THIS TESTS Whether you can reason about the two failure modes of queue consumers, a permanently bad message versus a temporarily unhealthy dependency, and apply distinct patterns to each.

A GOOD ANSWER COVERS For transient downstream failures, retry with exponential backoff and jitter so you do not synchronize a thundering herd against a recovering dependency. Cap the total attempts and total time. For poison pills, messages that will never succeed, enforce a maximum delivery count and then move the message to a dead-letter queue where it can be inspected, alerted on, and replayed after a fix. Make handlers idempotent, keyed on a stable message or business id, so a redelivery does not double-charge or double-send. Set the visibility timeout or lease longer than the worst-case processing time so a slow but valid job is not redelivered and processed twice. Consider a circuit breaker around the downstream so the whole consumer pauses when the dependency is clearly down.

COMMON WRONG ANSWERS Retrying forever, which lets one poison pill block the head of the queue and starve everything behind it. No dead-letter queue, so bad messages are silently dropped or loop. Non-idempotent handlers that corrupt state on retry. Fixed-interval retries with no jitter that hammer the downstream in lockstep.

LIKELY FOLLOW-UPS How do you tell transient from permanent. How do you alert on DLQ growth. How do you replay safely. What ordering guarantees break under retries.

ONE CONCRETE EXAMPLE A worker charges cards. The payment API returns 503 intermittently, so the worker retries up to five times with backoff and jitter and succeeds. One malformed message throws on every attempt, hits the delivery cap of five, and is routed to the DLQ, which pages the on-call when its depth crosses a threshold. Because the charge handler keys on order id, the earlier retries never double-charged the customer.