Queues vs Streams: When Should You Use Each?

By Arkaitz

You have something writing records in one service and something else needing them in another. The decision that follows is where you freeze. Kafka or RabbitMQ. SQS or Kinesis. The product names rush in before the question does. Queues vs streams is not a brand fight. The decision asks one question: does your message live once, as work to be done, or does the message live forever, as history to be read? Everything else follows from your answer.

Your design review cannot wait. If you want the wider frame first, read the system design practical guide; this post is the broker decision inside it. The rule here is simple, and the rest of the post earns it.

A queue deletes a message once a consumer acknowledges it; a stream keeps the message so any consumer can replay history later. Use a queue when work disappears once done. Use a stream when events are a product other services will need to read, now or later.

If your queue runs red and nobody knows why, jump to event-driven architecture for the decoupling frame and come back here for the mechanics.

What is the difference between a queue and a stream?

Choose by lifecycle, not by latency. A queue's message dies after ACK. A stream's message persists and anyone can read it later. That single difference determines replay, fan-out on history, and whether a new service can be built against old events.

Work vs fact: the lifecycle rule

A queue is a point of work. A producer puts in a message, one worker picks up the message, processes it, and acknowledges it. The broker then deletes the message. A queued message gets one consumer's attention and one ending.

A stream is a log of facts you can read later. A producer appends an event with a key and an offset. You and your consumers read at your own pace and your own position. The event stays, in order, for as long as your retention policy allows. New consumers can start at offset zero and read everything that happened before they existed.

So the real test is not do I need a broker. The test is whether the message's job ends at being handled, or whether its job is to become history. Messages that are commands, send this email, charge this card, resize this image, are work. Messages that are facts, order placed, payment captured, user signed up, are history.

When you classify a flow, ask yourself one question: if I deleted the message after one consumer handled it, would anything break besides that consumer? If the answer is no, you have work. If the answer is yes, you have a fact, and facts belong in a stream.

Take three common cases in your own codebase. An image resizing job: one caller asks for the resize, one worker does the resize, and nobody ever needs to know the resize happened again. That is a queue. An order placed event: billing, inventory, shipping, and analytics all need the event, and next quarter you will build a recommendation service that needs last year's orders. That is a stream. A webhook delivery: one attempt succeeds, the broker drops the message, and reading the message again would just double-send. That is a queue.

Most programs are either doing work for someone or broadcasting facts about what happened. Match the shape, not the brand.

Use case Nature of the message Pick
Task dispatch, image processing, email sending, scraping batches Work that disappears once done Queue
Fan-out to many consumers who need the full history Durable facts Stream
Workflow steps with retries, priorities, deadlines Work Queue
Change data capture, audit log, analytics ingestion Facts Stream
Request-reply between two known services Work Queue (or just call)
New service built against events that happened last year History needed Stream

You can still pick wrong by lifecycle if you guess. When unsure, default to a queue, because a queue is cheaper and simpler, and you can add a stream later when a second consumer actually asks for history.

If your system spans multiple services and you are sizing that boundary, read how to design a distributed system for the bigger frame and keep this post for the broker question.

When should you use a queue vs a stream?

Use a queue when a message is work that one worker must do; use a stream when a message is a fact that many readers may replay. Then cost-check against replay and fan-out, because many streams are queues in disguise.

The three positive triggers for a stream

You make the call in two questions: do you need history, and does more than one consumer need the event?

You do not pick a stream because the idea sounds modern. You pick a stream when at least one of these three triggers is true, and you accept the cost that follows.

First trigger: more than one consumer needs the same event and each must see all of it. Billing and analytics reading the same order event is the textbook case. A stream broadcasts to unlimited groups; a queue forces you to build per-consumer topics or an outbox.

Second trigger: a service that does not exist yet must read events that already happened. A new recommendation engine replaying a year of orders needs a log with retention, not a queue that deleted each message on ACK.

Third trigger: the event is the source of truth for state reconstruction. Event sourcing, audit logs, and change data capture all need the history to rebuild a view. A queue cannot rebuild anything; it only remembers what is left to do.

If none of those three triggers fires, use a queue. A queue scales workers independently, costs less at low volume, and keeps failure local. Workloads like sending email, charging cards, resizing images, and scraping pages all fit. Do not pay for replay and fan-out you will not use.

Picture a micro-SaaS billing service. It sends two things: invoice generated emails and subscription updated facts. The emails are work for one worker and a queue fits. The subscription facts feed billing, entitlements, and analytics, and next quarter customer success will want a export tool reading last month's changes. That second flow needs a stream. One product, two message lives, two brokers in the wrapper pattern at the end of this post.

Before you wire two brokers into a monolith, weigh the boundary cost against modular monolith vs microservices and monolith vs microservices. Broker choice is smaller than service boundary choice, and the wrong boundary will hurt more than the wrong queue.

Why does one slow message stall the whole partition?

What is head-of-line blocking?

On a stream, a slow worker blocks every message behind the slow worker in that partition for every consumer group reading it. On a queue, a slow message stalls only your concurrency for that queue; other workers keep pulling.

Head-of-line blocking is what happens when message N waits on a slow consumer and every message after N waits behind message N. On a partitioned log this is not a side effect; that behavior is the model. A Kafka consumer reads a partition in order. If message 7 takes 30 seconds because a downstream is slow, messages 8 through 700 wait in that partition until 7 finishes, no matter how many workers your group has. (Apache Kafka partition model).

A queue does not order within a worker pool by default. With a prefetch of one and per-message acknowledgement, a stuck worker just stops pulling; the broker redelivers the message to another worker after the visibility timeout. The cost is local: one lane, one slower job. The queue's ordering is delivery order among live consumers, which is a looser but safer shape for arbitrary workloads.

When does a slow message kill your throughput? Your throughput collapses when your task time varies by orders of magnitude. A scraping task that runs 50 ms against one site and 90 seconds against another, on the same partition, will stall the fast ones behind the slow one. Uber hit exactly this at scale: by routing more than a thousand consumer services through uForwarder, a push proxy, Uber decoupled partition count from processing concurrency, because Kafka consumer groups alone would cap parallelism at the partition count. (Uber Kafka architecture).

If your workloads are uniform and fast, a stream is fine. If your task times look like orders of magnitude, you want a queue, or a stream plus a queue downstream. Do not try to fix variable task times with more partitions; that only spreads the stalls.

How does fan-out differ between queues and streams?

One stream fan-outs to as many consumer groups as you want, all reading the same retained history at their own pace. A queue fans work out, one consumer at a time; two consumers on the same queue split the work rather than both reading it.

This is where queues vs streams stops being subtle. A stream is a broadcast of facts: add a new consumer group and it reads from the start of retention, or from now. Every group sees every event. Your audit service, your analytics sink, your cache invalidator, and your new microservice all run against the same log.

A queue is a work list: two workers on the same queue each get a disjoint subset of messages. That is what you want for do this work once. It is not what you want for everybody needs to know this happened.

Remember event-driven architecture: decoupling lives there; this post owns the broker mechanics. If you start with a queue and later need fan-out on history, you build it yourself: a topic per consumer, or a separate outbox per target. You take on the coordination cost. Decide up front whether the event is a product. If the event is a product, stream the event.

A practical test: list the consumers you know about today, then add the ones you will build in the next twelve months. If the list has more than two entries and they all need the same event, stream. If the list has one entry, queue. Revisit the choice only when a new consumer actually appears, because building for imaginary consumers is how Kafka clusters get filled with topics nobody reads.

What is the retention trap in stream replay?

Replay costs you retention. Short time-to-live values mean a failure nobody notices inside the window is permanent loss; queues offer no replay at all and cap retention at 14 days.

Replay is the feature that makes streams worth the cost. Kafka and similar commit-log stores keep data by time or size and let a consumer seek to any offset within retention. You pay for broker storage, or for a second raw copy in cheaper storage if the broker's retention is short. Netflix's production Kafka topics carried message time-to-live values as little as four to six hours, so a failure nobody notices inside that window is permanent loss; their mitigation was a second raw copy in HDFS, retained one to two days, purely for replay. (InfoQ's account of Netflix's pipeline).

A queue offers no replay. SQS adds no history at all: once a message is consumed and deleted, it is gone, and a new service cannot catch up. SQS and similar FIFO queues also cap retention at 14 days maximum. The message's lifetime is the work's lifetime.

That single difference is why the choice is often irreversible in practice. A stream lets a new team arrive and build against old events. A queue forces that team to start from today.

Run this as a runbook, not a guess. For every stream you own, write down three numbers: the retention window, the current consumer lag measured in hours, and the oldest unprocessed message currently visible. If consumer lag ever exceeds retention, you are already losing history and will not notice until someone asks where data went. Alert on lag, not on error rate, because a stream can be perfectly healthy by error rate while silently dropping history. If the runbook looks expensive, shorten retention honestly, and archive a raw copy to object storage for the rare replay.

If a backlog outlives its retention, the data is gone, and no replay fixes it. For the availability budgeting around that failure mode, hand off to how to design a highly available system.

So ask the uncomfortable question at design time: will any team we do not know yet need these events? If yes, stream. If no, queue, and accept that you save money and complexity for the features you actually have. If your answer is maybe, later, someone might, choose a queue now and revisit when that someone shows up.

What does a stream cost at small scale?

Queues are cheaper and simpler at low volume. Streams become cheaper per unit at high fan-out and high volume, but only if you actually replay and fan out. Otherwise you are paying broker-hours for a feature you do not use.

The cost model, not latency, decides most real choices. Managed queues like SQS bill per API request, so at low volume you pay cents for your own usage. Managed streams bill for broker-hours, partitions, and gigabytes in and out: on AWS, Kinesis on-demand carries a minimum account charge equivalent to 25 MB/s of ingest and 25 MB/s of retrieval even if you use zero, which is a real floor for small workloads. (Kinesis pricing). Self-hosted Kafka or its managed equivalent pays off above a certain throughput but demands on-call time from you: Netflix warned that adopting streaming early carries a pioneer tax in on-call burden and complex failure recovery, and Netflix said it outright: Don't stream all the things. (Netflix, quoted via InfoQ).

There is a cleaner proof of restraint. Netflix moved the Tudum fan site away from Kafka plus CQRS to an in-memory object store because read-heavy small state did not need a stream. The stream was the thing being paid for. The feature did not need it. (Netflix Tudum migration).

Run the cost against what you actually do, not against what you might. If you send 200 messages a second and have three consumers, SQS and a queue are fine. If you run millions of events a day with fifty consumer groups and frequent replay, a stream pays for itself. The crossover is not a fixed number; it is are you using replay and fan-out. If you are not, do not pay for them.

Use a 30-day bill test on your current queue or stream. Count messages produced, messages consumed per group, replays requested, and consumer groups active. Divide the broker bill by the number of features that would break if the broker vanished. If that ratio is large for one consumer and small replay, your broker is over-qualified. If it is small for many consumers, your broker is earning its place. Run this test before arguing about Kafka versus RabbitMQ; the numbers usually settle the argument before the opinions do.

One more cost nobody prices: on-call. A stream is a stricter alerting contract. AWS calls backlogs insurmountable when they outlive the broker's retention, and Netflix's short retention TTL means a stream failure nobody notices in time is permanent data loss. A queue failure is usually a delayed job. Pay attention to which failure you are buying.

How do FIFO ordering and throughput trade off?

Partition key vs prefetch vs FIFO message group

A single partition orders strictly by offset. A queue orders only within a message group, and only at prefetch one. Global order is a single-partition luxury, whichever broker you pick.

On a stream, you choose a partition key and every event with that key lands in the same partition, in order. Different keys go to different partitions and order between partitions is not defined. Order all my events globally means one partition, which means one consumer, which is the concurrency ceiling above.

On a queue, RabbitMQ-style consumers get messages in delivery order only when prefetch is 1. Raise prefetch to 10 and the broker sends 10 messages ahead; a worker that processes them out of completion order delivers results out of order, because the queue delegated ordering to your task scheduler. (RabbitMQ prefetch discussion). If you need strict per-entity order, keep prefetch at 1 or use a FIFO queue with message groups.

SQS FIFO gives you ordering per message group and exactly these throughput numbers: 300 TPS per group by default, 3,000 messages per second with batching, scaling up to 70,000 TPS and 700,000 messages per second with batching in high-throughput mode. (AWS SQS FAQs). That is a real ceiling you design against, and it is why FIFO ordering is something you pay for with throughput.

So the question to settle: order by what, for whom? If you need events for user 42 in the order they happened, a stream partition keyed on user 42 gives that order for free. If you need all events in the system in order, no broker gives that order at scale, and you should question whether the requirement is real.

When is Kafka a bad task queue?

The common one is running Kafka as a task queue. If your partition count is larger than your actual concurrency, your partitions stall and your throughput drops. Check that first.

Here is the checklist. Count how many apply and fix them in this order:

  • Kafka or another log used for task dispatch with highly variable task times, causing partition stalls.
  • One partition per worker just in case, leading to thousands of partitions and a broker that spends more time rebalancing than producing.
  • A queue adopted with no dead-letter queue, so one poison message blocks the whole queue forever.
  • Prefetch raised to 50 on a RabbitMQ consumer for throughput, silently destroying per-entity ordering.
  • A managed stream paying the on-demand floor for a workload that sends 50 messages a day.
  • Idempotency assumed because the broker guarantees it, when the guarantee stops at the broker's edge.

The hardest one to spot is the first, because the approach looks like squeezing efficiency out of one broker. The approach is not efficiency. The approach is a partition doing nothing while a slow task finishes. If you see stalled consumers and growing lag, you are looking at head-of-line blocking, and it traces to the partition model, not to your code.

If you recognized three of those, you are not alone. Most teams get here by adopting a broker before adopting the question. If your workload is mostly task dispatch with variable times, move it to a queue this quarter and keep the stream only for the facts.

What is the honest large-operator answer?

Wrap one inside the other: Kafka as the history backbone, SQS or RabbitMQ for task dispatch, joined by a small adapter. That is what the big operators run, and it is the honest production answer.

Do not treat the wrapper as a second-best fallback. It is the mature answer. A stream excels at history, replay, and fan-out; a queue excels at dispatch, elasticity, and task lifecycle. Trying to make one broker do both well is how you inherit the downsides of each without the upside of either.

The wrapper contract is small. The stream owns facts and retention; the adapter owns translation and ownership transfer; the queue owns work and retry. Keep the boundary explicit in your code so no service thinks it owns the other broker's concern. The adapter has three jobs: project a relevant fact to a task message, carry an idempotency key, and mark success only after the queue acknowledges enqueue.

Most systems arrive here by evolution, not by plan. Your system starts with a queue, adds a stream when fan-out demand appears, and keeps the queue for the work that never needed history. Do not jump to the wrapper before you feel the pain; do jump once you feel it, because by then you know exactly which side earns each job.

When should you start simpler than Kafka?

Do not stream all the things. Start with an in-memory store, or a single queue, and add the stream when a second consumer or a replay demand proves it.

Netflix moved the Tudum fan site away from Kafka plus CQRS to an in-memory object store because read-heavy small state did not need a stream. The stream was the thing being paid for. The feature did not need it. (Netflix Tudum migration).

Start simpler also means check your service boundary before you check your broker. If you are still one deployable, you probably need neither Kafka nor SQS; a database table with a status column and a worker pulling rows may be enough. When you split services later, the broker question comes back and this post gives you the rule. Read modular monolith vs microservices and monolith vs microservices before you buy a broker for a problem your boundaries created.

Your decision rule then: prove the need with one consumer and short retention, then invest. If no second consumer shows up in six months, your first answer was right and you saved yourself a cluster, an on-call rotation, and a repartitioning crisis.

What this post is not — and what to read instead

Queues vs streams is only the broker archetype. The bigger frame lives in sibling posts, and you do not need to re-read this article to make those decisions.

You now have the rule. When you apply this rule, you save money, and you keep your on-call quiet. Apply the rule before you buy infrastructure.

The thesis, one sentence: queues vs streams is a lifecycle question, not a brand question — a message that dies after being handled is a queue, and a message that becomes history for readers is a stream. If you are still sizing inter-service communication inside a wider design, read how to design a distributed system, and keep event-driven architecture next to it.

Related Reading

Back to all posts