Message Brokers

A message broker sits between services so they never have to be up, fast or synchronous at the same moment: producers hand it messages and move on; consumers take them at their own rate. It exists to absorb load spikes, survive consumer downtime, and cut the temporal coupling that makes chains of synchronous HTTP calls fragile — one slow service no longer stalls the caller.

Two shapes cover most needs. A queue (RabbitMQ, SQS) is for work distribution: each message — send this email, resize this image — is consumed once by one worker, then gone. A log (Kafka, Redpanda) is for event streams: an ordered, durable record that many independent consumers read at their own offsets, and can replay from the beginning — which is what makes it the backbone of event-driven architectures and analytics pipelines.

Delivery is at-least-once in practice, so duplicates are a design input, not an accident; exactly-once is either narrow or a marketing claim. Ordering holds only within a partition, and partitioning choices are hard to change later. And a broker is real infrastructure — clustering, retention, monitoring — which is why managed offerings are usually the right call, and why a plain database job table is often enough for small systems.

Pick the shape first: task distribution → queue; event stream with replay and fan-out → log. Make every consumer idempotent, route messages that repeatedly fail to a dead-letter queue instead of retrying forever, and monitor consumer lag — it is the earliest signal that consumers are falling behind. Keep payloads small and versioned; the broker outlives any one service's schema.