Event-Driven Architecture: When It Earns Its Cost

Somebody in your architecture review said the words out loud: we should go event-driven. Three posts before that made the same promise, and none showed you the bill. This one is the bill.

Event-driven architecture earns its cost only when you have real fan-out to consumers you do not control, a genuine need to decouple in time, or events that are a product in their own right. Everywhere else you are buying extra machinery, a stricter on-call contract, and a payload contract you now owe the whole company.

If you came here for a definition, close the tab; page one is nine vendor pages that stop at the benefits list. If you came here because a queue will not drain and someone in Slack is arguing that an event was lost, jump straight to the delivery guarantee.

You are reading one child of the system design handbook, the wider procedure for designing any system. What follows is the decision that sits under the handbook: five audits of the choice to run your backend on events—coupling, write path, delivery guarantee, failure modes, and your team.

Your running example is Sentry, whose own engineers document the pipeline in public, inside a monolith. The parts the marketing pages skip are the parts that price the decision for you.

What are you actually adopting when you go event-driven?

You are adopting a set of coupling decisions, not a product. Event-driven architecture is about who knows about whom, and when. The broker is the cheap part; you pay for everything downstream of that sentence.

Your most honest starting point is still Martin Fowler's map, because it names the four different things people mean when they say event-driven: event notification, event-carried state transfer, event sourcing, and CQRS (What do you mean by Event-Driven?). You will probably reach for the first and quietly buy the fourth.

Keep one distinction from the map: an event is a past fact you cannot argue with, and a command is a request exactly one party must honor. That line decides who owns the next step.

The standardization that exists covers the envelope, not the promise. CloudEvents gives you a common shape for describing event data (cloudevents.io, a CNCF project that graduated in January 2024), and it says nothing about whether your event arrives, in order, or once.

And here is what the vendor decks leave out. Sentry, an open-source project active since 2010 and now on a source-available license, is still shipping commits (the repository), and it calls itself a monolithic Python Django application deployed in several form factors (web servers, Kafka consumers, Celery workers, etc), held together by a transactionally written event queue (aka an 'outbox') (A $3,000,000 Dropdown). So event-driven architecture is not a service-splitting decision; that argument belongs to the modular monolith middle.

When does event-driven architecture earn its cost?

Use event-driven architecture when you have real fan-out to consumers you do not control, when you genuinely need to decouple in time, or when the events are a product in themselves. Do not use events for a straightforward request-response interaction, for a transaction that needs strong consistency across services, or before your team can operate asynchronous systems at all.

That is the whole decision, and it is the same three triggers every other guide lists. What changes your answer is the receipts underneath.

You meet Microsoft's Azure Architecture Center's refusals first: a straightforward interaction where a synchronous call already meets your latency and throughput needs, a business transaction that demands strong consistency across services, and a team with no experience running asynchronous systems (Event-Driven Architecture Style). If your system is all three, you have your answer, and the answer is not events.

Tape Gregor Hohpe's warning to the wall: the topology decoupling payoff is less-pronounced in your own application, where you control both sender and receiver (Event-driven = Loosely coupled? Not so fast!). In your own system, the benefit you were sold is the benefit you get least.

Netflix learned this the expensive way and says so. The same case study that reports 450 billion unique events a day also warns against streaming all the things and names a pioneer tax for teams who adopt streaming early (InfoQ's account of Netflix's migration). The technology was fine. The bet was the maintenance.

And the bill scales with the number of consumers in a way nothing shows you at the start. Adding a subscriber is nearly free; owning a subscriber is not. Hohpe's conclusion is blunt: having dependencies between components all over the system is likely a poor design choice and should not be considered 'loose coupling.' Cheap to add, expensive to own.

If you are at 100 users, none of this applies yet. This is a decision for systems that have outgrown one process, and the honest starting point for everything below is the smallest architecture that still works for you.

Why async equals decoupled is wrong

Async removes exactly one kind of coupling, the requirement that both sides be up at the same moment, and leaves every other one standing. Audit your coupling by facet, because events fix some and quietly worsen others.

Hohpe's 2024 piece lists the facets you have to think in: temporal, location, space, topology, and data format. Asynchronous one-way messaging kills temporal coupling and location coupling, because you no longer address a live endpoint, and it does nothing for data format.

Keep one Hohpe sentence in mind: Most of the decoupling properties of EDAs derive from the use of Publish-Subscribe Channels, not event semantics. A command on a publish-subscribe channel is as decoupled, topologically, as any event. Event-driven is not where the benefit lives. Publish-subscribe is.

The five facets of coupling

Async removes temporal coupling and does nothing for data-format agreement, so your sender and receiver stay coupled on the schema. That is the coupling that outlives the win, and here are all five:

• Temporal — must both sides run at once? Async one-way messaging removes this one.
• Location — do you know where the other side is? Removed too.
• Space — do you address the other side by name? A topic is looser than a hostname, but still a name.
• Topology — can you add a participant without changing anyone else? This is the facet the marketing sells.
• Data format — do you agree on the payload and the meaning? Events never fix this one.

Topology is the facet where your broker can quietly negate the very decoupling you bought. EventBridge-style rules-and-targets behave as a Recipient List: adding a recipient means changing a central element, so change propagates in both directions, the exact property publish-subscribe was supposed to remove. In your own application, says Hohpe, adding recipients free of side effects is a less-pronounced benefit.

Fowler's follow-up is the other half. When a logical flow runs over a chain of event notifications, it can be hard to see such a flow as it's not explicit in any program text. Nothing in your repository says step three happens after step two. The flow lives in the wiring, and the wiring is not code you review.

And this is the price no benefits list quotes: loose coupling costs you compile-time validation. The line is in Hohpe's W-JAX 2007 talk, and the wording has not aged: Integration testing more critical – less compile time validation (the price of loose coupling) (Hohpe's W-JAX 2007 talk). Your type checker cannot see across a broker.

How do events actually leave your database?

Write the state change and the event in one local ACID transaction, into an outbox table inside the same database, then let a relay publish the event. The two writes are atomic because you do them as one write. Skip the outbox and you have a dual-write gap.

Where state lives inside your system, and how events leave, is the ordered sequence how to design a distributed system walks through; this section is the one audit inside that sequence that the event-driven decision owns.

Debezium names the problem you already suspect. Your service writes your local data store, then wants to publish an event, and those are two separate steps that can succeed or fail independently. A crash between those two steps leaves your database and your broker telling different stories—that is the issue of dual writes (Reliable Microservices Data Exchange With the Outbox Pattern).

The outbox makes the event part of the same local transaction; Chris Richardson's pattern page names the pattern and its alias, Application events (Transactional outbox).

This is standard tooling, not folklore. Debezium ships a supported transformation for exactly this, and the documentation says so plainly: the pattern avoids inconsistencies between the service database and the broker (Outbox Event Router). Thorben Janssen's implementation guide walks the publisher half end to end, and it is the same three moves every time (Implementing the Outbox Pattern with CDC).

You already met the production proof. It is not a distributed system of a hundred services. Sentry is one Django application in several form factors, and it still needed a transactionally written event queue to keep the pieces consistent.

Event-driven architecture is not a microservices prerequisite. It is a decision about atomicity at every boundary where you write, whether or not that boundary is a network call.

CDC or polling — the two relay choices

Pick change data capture for near-real-time and low load, or polling for simplicity and no database privilege change; the pattern is the same either way. CDC reads the database's own log and turns each committed row into an event; a poller runs a query and marks rows as sent. Which of the two you run, and every mechanism behind it, is queues-versus-streams territory, named here once and deferred to that post.

The one thing to get right regardless of relay is deletion. Events accumulate, and you have to bound what you keep, which is the retention problem, and retention design is also not this post.

What delivery guarantee do you actually have?

Plan for at-least-once everywhere and make every consumer idempotent, because every broker's exactly-once stops at its own boundary. No broker promises exactly-once across your whole system, and the good ones admit it in writing.

You get the most quotable refusal on the page from Kafka's own documentation: Many systems claim to provide 'exactly-once' delivery semantics, but it is important to read the fine print, because sometimes these claims are misleading (Apache Kafka design). The docs then split the claim into publishing durability and consuming, which are two different problems with two different answers.

Amazon SQS says the same about standard queues: they ensure at-least-once message delivery, but due to the highly distributed architecture, more than one copy of a message might be delivered, and messages may occasionally arrive out of order (Standard queues). NATS JetStream adds persistence and at-least-once on top of core NATS, and Jepsen's analysis records the promise plainly: messages may be duplicated, but acknowledged messages should not be lost (Jepsen: NATS 2.12.1). RabbitMQ gives you publisher confirms as the broker-side acknowledgement and prefetch as the bound on your unacked window, and those are exactly the knobs that make duplicates your responsibility (Consumer Acknowledgements and Publisher Confirms).

The exactly-once fine print

Exactly-once is an in-broker claim. Kafka's idempotent producer and Kafka's transactions stop duplicate writes into Kafka, not duplicate effects in your consumer. Idempotent producers and atomic sends to multiple partitions have shipped since version 0.11.0.0, and the guarantee stops at Kafka's edge.

Underneath even a correct guarantee sits a durability policy you did not read. Jepsen tested NATS 2.12.1 and found committed writes lost and a persistent split-brain after coordinated power failures, caused at least in part by choosing to flush writes to disk every two minutes rather than before acknowledging them (Jepsen: NATS 2.12.1). The same analysis points at the documentation, which claims both linearizable and self-heal and always be available, two things that cannot both hold. That is the CAP theorem's subject, not this post's (the CAP theorem post).

So you build the consumer side yourself. The Enterprise Integration Patterns catalog calls it the Idempotent Receiver: a receiver must tolerate a message being received once or multiple times, either by de-duping explicitly or by defining the message semantics to make repetition harmless (Idempotent Receiver).

The API designers show you how it feels. Stripe auto-prunes idempotency keys after at least 24 hours, and a key reused after pruning silently becomes a new request (Idempotent requests). AWS returns a parameter-mismatch error when a known token arrives with different parameters, which protects the service but leaves the caller uncertain whether this request did the work (Making retries safe with idempotent APIs).

And the reliability mechanism you add for failures is a duplicate source too. RabbitMQ's at-least-once dead-lettering exists precisely because the traditional dead-letter-exchange approach has not been safe in a cluster, and the fix carries its own warning about a possibility of duplicates appearing at the DLX target queue(s) (Quorum Queues, At-Least-Once Dead Lettering).

What kills event flows in production?

Poison messages that block a consumer forever, retry storms that multiply load across every consumer group, and backlogs that outlive the broker's retention and become permanent data loss. Design the death paths, bounded retries and dead-letter queues and backpressure, before you design the happy path.

The retry arithmetic is the part you underestimate, because the arithmetic is the same as RPC and you assume events dodge it. Events do not.

AWS's Builders' Library states it flatly: retries are selfish, and if each of five layers in your stack retries three times, load on the failing dependency rises by a factor of 243 (Timeouts, retries, and backoff with jitter). Google's SRE book gives the same shape in the other direction: three layers with four attempts each turns one user action into 64 attempts, and the retry-budget example is 60 retries per minute, per process (Addressing Cascading Failures). AWS's fix is blunt, and it applies to you: retry at a single point in the stack, and nowhere else.

Events multiply that arithmetic, because each consumer group carries a separate retry policy independently. Five layers with three retries each is bad; fan that across four consumer groups, each with a separate policy, and no single owner sees the total.

Then there is the backlog. AWS documents backlogs that are insurmountable, meaning unrecoverable in a reasonable time, and it gives you exactly four levers: backpressure, prioritization, sidelining excess traffic to a separate queue, and constant work to drain (Avoiding insurmountable queue backlogs). Netflix shows you what happens when the backlog outlives retention: Netflix's production Kafka topics carry message time-to-live values as little as four to six hours, so a stream failure nobody notices inside that window is permanent loss, and the mitigation was a second raw copy in HDFS, retained one to two days, purely for replay (InfoQ).

That last one is a phrase worth sitting with: a queue is not a buffer. A queue is a bet on your alerting.

The four levers for an insurmountable backlog

Bound intake, prioritize what must drain, sideline excess to a separate queue, and do constant work, because a backlog that outlives your TTL is data loss, not latency. Dead-letter queues are the standard escape hatch when a message cannot be processed at all, and the pattern catalog names dead-letter queues for a reason; depth and replay and redrive belong to the queues-versus-streams post.

Choosing streaming is also choosing a stricter on-call contract, and Netflix states the difference better than any SLO doc: batch failures have to be addressed urgently, streaming failures have to be addressed immediately (InfoQ). That is a real cost, paid in nights and weekends.

The availability budgeting behind all of this, targets and error budgets and failover, is a different post (how to design a highly available system). This section owns only the failure mode that is specific to events: lag and time-to-live, and the fact that both are correctness inputs, not performance knobs.

Choreography or orchestration — and who owns the flow?

In choreography, each service reacts to events and knows the next step; no component owns the flow, which makes it hard to see and debug. In orchestration, one component holds the state and tells the others what to do. Use choreography only while the flow fits on a whiteboard and needs no compensation.

You get the cleanest description of the trade from Azure's two topologies. In the broker topology there is no central coordination or orchestration, so this topology can be dynamic and highly decoupled, and no component owns or is aware of the state of any multistep business transaction—which is why the docs warn you to carefully consider error handling and manual intervention strategies because this topology can be a source of data inconsistency (Event-Driven Architecture Style). In the mediator topology one component maintains the state and manages error handling and restart capabilities, dispatching commands to designated channels. You have traded decoupling for a thing you can watch.

The moment a flow needs compensation, the choice is made for you. A saga, the concept from Garcia-Molina and Salem's 1987 work, is a long-lived transaction broken into subtransactions, each with a compensating action that undoes the earlier step if a later one fails (the 1987 report). Compensations are stateful and order-dependent, and something has to own that state. Choreographing compensation across five services by hand is how a whiteboard flow becomes undiscoverable.

Akka's own saga series is the field report. Choreography-based sagas are, in Akka's words, distributed event-driven workflows, and the same series devoted a whole installment to dealing with poison messages and ensuring that a single failed event doesn't block the entire process (Saga patterns in Akka, part 5). The hard part arrived in the failure path, which is the pattern.

And here is the counter-intuitive part, which is Fowler's. When a logical flow hides inside event notifications, the flow is not explicit in any program text, and often the only way to figure out this flow is from monitoring a live system. Your code review cannot see the flow. Your diagram cannot either, once the diagram drifts. Only production can, and only if you instrumented the path.

The pragmatic answer is usually a hybrid, and Sentry is that too, using RPCs and a transactionally written event queue together: the synchronous call where a call is what you mean, the event where fan-out is what you mean. The phrase pinball architecture describes the choreography-gone-wrong case, but treat the phrase as folklore; no citable origin for the term was verified.

How do you evolve an event schema without breaking strangers?

Treat the event schema as a published API: add optional fields, never rename or delete in place, and let a compatibility mode be the gate. You do not know every consumer, so you cannot coordinate the change, which means the schema is the one contract you must treat as permanent.

Schema resolution is the mechanism that lets this work at all. Avro defines how a reader schema reconciles data that was written with a different writer schema, which is what lets one consumer upgrade while another does not (Apache Avro specification). Fowler's point stands over the top of the mechanism: once you are event-driven, multiple working copies with different schemas over one event log is normal, not a mistake. You are not upgrading a database; you are upgrading a pub-sub system, where every subscriber upgrades on its own schedule.

This is the data-format facet from earlier, arriving with a bill: the coupling that survives the async win. Treat the schema with the care you would give a public REST contract, because that is what it is.

CloudEvents gives you the envelope, and only the envelope. It standardizes the shape of the wrapper, not the payload inside it and not the promise around it. An envelope is not a guarantee.

Thin notifications or event-carried state transfer

Thin notifications couple consumers to your read API, because the consumer has to call you back for the details. Fat events couple every consumer to your payload schema. Pick per event, and version either one. There is no free option; there is only which coupling you prefer for this event.

Reach for Hohpe's own lesson from a first-hand project: The event classes are the primary coupling points between calculators (Look Ma — No Middleware!). A fat, behavior-rich event turns one team's refactor into everyone's incident, because everyone now depends on the shape of a thing they do not own.

Two sibling topics get named once and left alone. Event sourcing is storing the event log as the source of truth; CQRS is separating the read model from the write model. Whether you actually need either is a different question, and Debezium's own comparison argues that change data capture plus an outbox is usually the better answer for exchange between services (Event Sourcing vs. Change Data Capture).

How do you observe an event-driven system?

Propagate a trace context through every hop, treat consumer lag as a correctness signal rather than a metric, and alert on dead-letter depth, not just on errors. The whole point of decoupling in time is that the other side does not have to be up, and that is exactly what makes the other side's silence invisible.

One NATS line puts the observability problem in front of you: JetStream extends that decoupling to time, the two no longer need to be online at the same moment (JetStream). A consumer that is down looks identical to one that is fine with nothing to do, and nothing on your error dashboard tells you the difference.

You have to know your own pipeline well enough to see the whole path, and Sentry's own answer is to publish an interactive map of the event ingestion path so the whole company can reason about the flow (Sentry: Event Ingestion Pipeline). That is the standard: if you cannot draw the path, you cannot alert on it.

This is deliberately short: deep observability mechanics, dashboards and lag curves, are general material, and the general design procedure owns that ground. The two signals specific to events are lag as correctness and dead-letter depth as a first-class alert.

How do you adopt EDA without a rewrite?

Pick one flow, put the event behind your existing database transaction, and grow the event surface the way Sentry did, inside the application you already run. You are adding a parallel path, not replacing the old one, and the old one keeps working until you retire it on your own schedule.

Hohpe wrote the whole strategy up from a first-hand project: Add rather than replace (Look Ma — No Middleware!). Run the event path next to the call path, move one consumer at a time, and keep event replay as your recovery tool when a consumer's state turns out to be wrong.

Sentry's shape is the template. One Django application, several form factors, an outbox inside the transaction, and no rewrite. You start with the flow where fan-out is real and the write is already atomic, and the outbox is off-the-shelf plumbing rather than something you invent (Outbox Event Router).

The far end of the same road is a streaming-first backbone, and Monzo is the honest example: most of our data is in the form of event data being produced in our backend services, streamed via the firehose using Kafka and NSQ into append-only projects in BigQuery (An introduction to Monzo's data stack). That is a regulated bank whose backbone is events, and it is the destination, not the starting move. You get there one flow at a time.

One selection rule belongs down here and nowhere else: a queue distributes work, a stream distributes events, and a stream with independent consumer groups is a durable broadcast. Every mechanism behind that sentence, partitions and retention and consumer-group choreography, is the queues-versus-streams post's job, not this one's.

What this post is not — and what to read instead

Queues versus streams, event sourcing, CQRS, CAP and the consistency-model taxonomy each have a dedicated post; this one ends where the others begin. Five audits, one clause each, so you can carry the list into the design review.

• Coupling. Async fixed the temporal facet and left your schema coupled. Budget for the schema.
• Write path. State change and event in one transaction, relayed from an outbox, or you have a dual-write gap.
• Guarantee. At-least-once everywhere, idempotent consumers, and a broker's exactly-once that stops at its own edge.
• Failure modes. Bounded retries, four backlog levers, and dead-letter depth you actually watch.
• Adoption. One flow, inside the system you already run, added rather than replaced.

The siblings, each named once: queues versus streams owns the broker and partition mechanics; event sourcing and CQRS own those patterns; consistency versus availability and strong versus eventual consistency own the taxonomy; and how to make better architecture decisions owns the record-keeping. This post owns the decision to go event-driven and its consequences, and nothing above reopens into that territory.

And the honest refusal this whole post rests on: if you are still at 100 users, none of this applies yet. Read ecommerce architecture at 100 users, run the smallest architecture that works, and come back when the fan-out is real and you own consumers you cannot see.