How to Design Reliable AI Systems: A Reliability Sequence

The demo works. The agent answers the question, calls the right tool, and produces a tidy summary. Then you put it in front of customers, and within a week it has invented a refund policy, retried a payment twice, and escalated a routine ticket to a human who did not have the context to resolve it.

The usual response is to look for a better model. That is the wrong first move. A stronger model raises the ceiling on what the system can attempt. It does not tell you whether an attempt succeeded, stop a duplicate side effect, or bound the cost of a runaway loop. Those are system properties, and you have to design them.

I treat AI reliability as a sequence of six decisions, and I would rather ship a boring fixed workflow than an impressive agent I cannot verify. The sections that follow take the decisions in order, with the eval and hand-off sections carrying the measurement and failure-path steps: bound the job, define success, verify the outcome, control side effects, cap consumption, and design the failure path. Get those right and a mediocre model can be dependable. Skip them and the best model on the market will still produce a confident incident.

What does AI reliability mean for a probabilistic system?

Reliability is the probability that the system completes the intended task correctly, within its cost and latency budget, and fails safely when it cannot. That definition has four parts, and the model only directly influences the first.

It also gives you something you can measure. The agent feels good is not a reliability target. 92% of refund requests resolve correctly, with 0 duplicate refunds, under €0.04 and 8 seconds at p95, and every unresolved case escalates with context is a target. You can build a dashboard, an alert, and an eval suite around that sentence.

The mistake is to treat reliability as a property of the model card. Model benchmarks measure capability on a fixed task distribution. Your system has a different distribution, a different tool surface, different data, and different consequences. The benchmark can tell you the model is capable of the task. It cannot tell you your system is reliable at it.

That distinction is why the definition has to be operational. A model card reports an average over a benchmark distribution; your system has a different distribution, a different failure cost, and a different definition of done. Write the success sentence for your system, then instrument the four parts. If you cannot measure one of them, you have found the first gap in the design.

Why do AI systems fail more than the demo suggests?

Because errors compound. A single step that succeeds 95% of the time looks excellent in isolation. Chain twenty of those steps and the probability that all of them succeed is 0.95²⁰, about 36%. The demo usually shows one or two steps, with a human quietly correcting the rest.

The lab data points the same way, at system level rather than step level. The MAST study annotated more than 1,600 traces across seven multi-agent frameworks and found failure rates between 41% and 86.7%. The taxonomy groups 14 failure modes into three categories: system design, inter-agent misalignment, and task verification. (MAST study) Notice what dominates. The failures are not mostly the model was not smart enough. They are specification gaps, coordination breakdowns, and missing verification.

That finding should change where you spend your effort. If most failures are specification, coordination, and verification failures, then more prompt tuning is a low-yield investment. The high-yield work is making the task narrower, the handoffs explicit, and the success check independent of the model that produced the output.

There is a second reason failures surprise teams: the failure is often plausible. A wrong answer reads like a right one. A duplicate payment looks like a successful payment in the UI. A skipped verification step produces a confident summary. Reliability engineering for AI is largely the discipline of making the system prove what happened, rather than asking you to trust the narrative.

How do you bound the job before adding autonomy?

You bound the job by choosing the least autonomous design that solves the problem. Anthropic's guidance on building effective agents makes the same point: start with a single augmented model, add a workflow when the steps are known, and add an agent only when the path genuinely cannot be fixed in advance. (Anthropic)

The agent loop, memory, and orchestration details belong in designing AI agents; here I care only about the reliability consequence. That gives you a ladder:

A single call with retrieved context. Best for classification, extraction, summarisation, and question answering over a known corpus.
A fixed workflow. The steps are known; the model fills in the decisions inside each step. Easier to test, cheaper, and much easier to debug.
An agent with a bounded tool set. The model chooses the path, but the environment, the tools, and the stopping conditions are constrained.
A multi-agent system. Multiple planners, handoffs, and shared state. Highest capability ceiling, highest coordination cost, and the setting where the MAST failure modes concentrate.

Every step up the ladder buys flexibility and pays for it in verification difficulty. Most production use cases stop at step two and call it an agent because the word is fashionable. That is a mistake in the other direction: if the path is fixed, do not pay for a planner.

The practical test is simple. Write down the sequence a correct run must follow. If you can write it without branching, you have a workflow. If the branches depend on information the system discovers at runtime, you need an agent, and you need to define what the agent is allowed to discover and do.

How do you measure an AI system that changes every run?

I measure outcomes with an eval suite, and I accept that a single run proves almost nothing. Anthropic's eval guidance is the most practical starting point I have found: begin with 20–50 tasks drawn from real failures, use deterministic graders wherever possible, add calibrated model-based graders where the outcome is subjective, and track consistency across trials rather than one lucky pass. (Anthropic)

The last point is the one teams skip. pass@k measures whether at least one of k attempts succeeds. pass^k measures whether all k attempts succeed. For a customer-facing agent, pass^k is the honest number: a 75% per-trial success rate passes three consecutive trials only about 42% of the time. If your product promise is it works every time, measure the every-time number.

Start with 20–50 real failures

Do not build a synthetic benchmark first. Go to the support queue, the incident tracker, and the review comments. Take the cases where the system was wrong, slow, or confusing. Those cases have three advantages: they reflect the real distribution, they come with context, and someone already cares about the outcome.

Write each task so two domain experts would agree on pass or fail. Ambiguity in the task becomes noise in the metric. If the task needs a reference solution, write one; it proves the task is solvable and catches broken graders.

Grade the outcome, not the path

It is tempting to assert the exact tool calls and the exact order. Resist it. Agents find valid paths you did not anticipate, and a path-based grader punishes them for being creative while missing the cases where they produce the right answer for the wrong reason.

My default is to grade the final state. Did the refund exist in the ledger with the correct amount? Did the ticket close with the right resolution code? Did the generated answer cite a source that supports every factual claim? Use transcript checks for safety properties, such as never called the delete tool, not for style preferences.

Then run the suite on every prompt change, every model upgrade, and every tool change. Evals are not a pre-launch ritual. They are the regression test that lets you adopt a new model in days instead of weeks.

How do you make success verifiable?

You give the system an independent way to check its own work. The check should not be the same model that produced the output, and it should not be a sentence in the prompt asking the model to be careful.

There are four useful verifiers, in descending order of trust:

Deterministic state checks. A test suite passes, a row exists, a balance matches, a schema validates. If the outcome can be checked by code, check it with code.
Policy and rule engines. The output must satisfy constraints you can encode: allowed refund range, permitted jurisdictions, required fields, banned operations.
Retrieval grounding. Every claim must cite a retrieved passage, and a separate check confirms the passage supports the claim. This is where production-ready RAG systems earn their keep: the retrieval layer is also the evidence layer.
A calibrated model judge. Useful for tone, completeness, and open-ended quality, but only after you have compared its judgments with human judgments on a sample.

The ordering matters. Teams often reach for an LLM judge first because it is easy, then discover that the judge is non-deterministic, expensive, and occasionally wrong in the same direction as the model under test. Use the judge where code cannot express the criterion, and calibrate it against humans before you trust it.

A verifier also gives you a stopping condition. An agent with no verifier does not know when it is done, so it either stops too early or loops.

Do not stop until the tests pass is a reliable instruction only when there is a test.

How do you make side effects safe?

You assume the model will retry, duplicate, and misorder actions, and you make the system tolerate it. This is ordinary distributed-systems discipline applied to a new caller.

The retry trap

A timeout does not mean the operation failed. It means you did not receive the response. If the agent retries a create operation after a timeout, you may create two records. Stripe's API solves this with idempotency keys: the client sends a unique key, and a retry with the same key returns the original result instead of performing the operation again. (Stripe)

This is the same at-least-once problem you solve in a distributed system design, now with a model choosing when to retry. Every write tool in an AI system needs the same property. Generate the key before the first attempt, persist it with the task, and reuse it on every retry, within the key's retention window. Do not let the model invent the key; it will invent a new one.

The same logic applies to event-driven consumers. A consumer that can receive the same message twice must be idempotent, or the duplicate becomes a business incident. An agent is just another consumer with a more creative reason to retry.

Least privilege and approval

I give the agent the smallest capability that completes the task. A support agent that needs to read an order does not need a database role that can update every table. A coding agent that needs to run tests does not need production credentials.

Then separate reversible from irreversible actions. Reading, drafting, and computing are reversible. Sending, paying, deleting, deploying, and changing permissions are not. The second category needs an explicit approval step, and the approval must show the actual arguments, not a friendly summary generated by the same model that proposed the action.

OWASP's excessive agency entry names the three ways this goes wrong: too much functionality, too many permissions, and too much autonomy. (OWASP) The fix is not a better system prompt. It is a smaller tool surface, scoped credentials, and a human gate where the blast radius justifies it.

The liability does not disappear because a model wrote the answer. In Moffatt v. Air Canada, the tribunal rejected the airline's argument that its chatbot was a separate legal entity and held the company liable for negligent misrepresentation. (Moffatt v. Air Canada) Your approval design is a liability design, not only a UX choice.

A sandbox is the other half of the control. Run generated code and untrusted tools in an environment with a filesystem boundary, a network allowlist, and a time limit. If the agent can reach the production network, the sandbox is decorative.

When should a human be in the loop?

I put a human in the loop when the action is irreversible, when the request is ambiguous, or when the system's confidence is below the threshold you set for autonomous action. Remove the human when none of those are true, because a review step that adds twenty minutes to every routine case is not a control; it is a queue that trains reviewers to click approve.

The design question is where the human sits. There are three useful positions:

Before the action. The human approves the plan or the exact call. Best for money, deletion, deployment, and permissions.
After the action, before the consequence is visible. The human reviews a draft, a refund, or a customer response. Best when the system can prepare work but the cost of a wrong send is high.
On exception. The system runs autonomously inside a policy and escalates only when a rule fires. Best for high-volume, low-variance work.

The third position is where most mature systems end up, but only after the first two have generated enough data to define the policy. Skipping straight to exception-only handling is how you get an agent that confidently handles the cases it should have escalated.

Whatever the position, the escalation must carry context: the original request, the tool calls already made, the evidence collected, and the reason for stopping. A handoff that says the agent could not complete this is worse than no handoff. The human starts from zero, and the customer waits.

How do you stop cost and latency from becoming the outage?

You stop cost and latency from becoming the outage by capping turns, tokens, wall-clock time, and spend before launch, then breaking the circuit when a dependency fails. A loop that never terminates is an availability incident with a bill attached. OWASP names the user-driven version of this risk unbounded consumption: a caller can force uncontrolled inference until the bill or the service gives out. (OWASP) The same failure arrives from inside the system as a retry storm or a recursive tool call.

I set the limits before launch:

Turn limit. A hard cap on model calls per task. When it is reached, stop and escalate.
Token budget. A per-task cap on input and output tokens. Track it per tenant, not just globally.
Wall-clock timeout. A deadline for the whole task, not just each call.
Cost ceiling. A per-task and per-customer spend limit that triggers an alert, then a block.
Rate limit. Protect downstream APIs from the agent's retry behaviour.
Circuit breaker. When the model provider or a tool fails repeatedly, stop calling it and degrade.

These are the same controls you would put on any unreliable dependency, which is what the model is. The difference is that the model's failure can be expensive in tokens as well as time. A retry loop with a 100k-token context can burn a month of budget in an afternoon.

There is a latency version of the same problem. An agent that makes six sequential model calls at two seconds each takes twelve seconds before it does any work. Parallelise independent calls, cache stable results, and route simple classifications to a smaller model. Reliability includes showing up on time.

What does the operating model look like?

You operate an AI system the way you operate any production service: with traces, service-level indicators, canaries, and an incident process.

Trace every step. Record the prompt, the model, the tool calls, the arguments, the result, the latency, and the token count. Without the trace, you cannot reproduce a failure, and you cannot tell a model problem from a tool problem. Treat the trace as sensitive data: it contains user input and possibly secrets.

Instrument the outcomes, not just the calls. Track task success rate, escalation rate, duplicate-action rate, cost per successful task, and p95 latency. A model call that succeeds is not a task that succeeded.

Canary every change. Prompt changes, model upgrades, and tool changes all go to a small slice first, with the eval suite and the production metrics compared against the baseline. Roll back automatically when the error budget burns too fast.

Review incidents with the same rigour as any other outage. The production incident triage pipeline applies: reconstruct the timeline, find the contributing causes, and add a regression case to the eval set. The difference is that the root cause is often a missing verifier rather than a bad line of code.

Watch for drift. User behaviour changes, upstream APIs change, and the model provider updates the model behind an alias. A suite that passed last month is evidence about last month. Re-run it on a schedule and after every provider change.

The availability discipline is familiar too. A dependency you cannot control needs a fallback, a timeout, and a clear degradation path, which is exactly the argument in how to design a highly available system. The model is a dependency with a natural-language interface. Treat it like one.

What should you build first?

I start with the smallest version of each control:

Write the success sentence. One sentence with a correctness criterion, a cost ceiling, a latency target, and an escalation rule.
Collect 20 real failures. Turn each into a task with a reference outcome and a deterministic check where possible.
Add one independent verifier. Prefer a test, schema, or state check over a model judge.
Make every write idempotent. One key per task, generated before the first attempt, reused on every retry.
Set a turn, token, and time limit. Test what happens when each one fires.

None of these requires a new model. All of them make the next model upgrade safer, because you will be able to tell whether it helped.

Reliability is not the absence of mistakes. It is the presence of a system that notices them, contains them, and tells you the truth about what happened. The model can be the least predictable component in that system. It should not be the only one holding the guarantees.