Designing AI Agents: An Architectural Guide

An agent is a control loop over tools. Everything past that sentence is packaging. Something reads state, decides on an action, calls a system outside your process, observes what came back, and repeats until a stop condition fires.

Anthropic's engineering team landed in the same place after a year of customer work. What held up was not elaborate frameworks. What held up was LLMs using tools based on environmental feedback in a loop, with simple designs and well-documented interfaces (Anthropic, Building effective agents). OpenAI's guide says the quiet part out loud: this concept of a while loop is central to the functioning of an agent (OpenAI, A practical guide to building agents).

So treat this as a systems problem, not a prompting problem. You can change a prompt in an afternoon. The loop decides whether a bad afternoon becomes an incident.

What follows is the architecture around that loop, in the order I would build one: the loop, the state behind the loop, the contracts your tools honour, the sandbox your tools run in, where you put a human sign-off, the traces you keep, the harness that grades a run, and the ceilings that stop one bad run from becoming an expensive one. Then the four failure modes that survive every one of those layers.

Designing AI agents is control-loop design, not prompt design

An agent is a bounded loop with a model in the decision seat, and your architecture is everything that keeps that loop bounded, observable and reversible.

The loop has a paper. ReAct interleaves a reasoning trace with an action, feeds the observation back in, and repeats. The authors report that reasoning traces let a model induce, track, and update action plans as well as handle exceptions, while actions let the model reach for facts outside the loop (Yao et al., ReAct).

That is the whole mechanism. Neither the reasoning text nor the tool call is magic: the artefact is a transcript you can read and a state you can inspect.

Google's pattern catalogue is more useful than most, because it sorts patterns by whether you can know the path in advance (Google Cloud, agent design patterns). Deterministic work — sequential steps, parallel fan-out, iterative refinement — needs no model orchestration at all. Dynamic work is where a single agent, a coordinator, a hierarchy or a swarm earns the complexity. Anthropic draws the same line in different words: workflows orchestrate models and tools through predefined code paths, while agents let the model dynamically direct their own processes and tool usage.

That is your first architectural decision, and you should make it deliberately. If you can know the path, build the workflow. A workflow produces the same trace twice, and a trace you can reproduce is a trace you can test. Google's guidance is blunter still: where the workload is predictable, or one model call handles the job, a non-agentic design is cheaper.

Choose the agent and four things become yours:

• Termination. Nothing inside the model guarantees the loop stops; your orchestrator does.
• Authority. Every tool you attach is a capability you granted.
• Cost. Each step is a billed call plus a network round trip.
• Explanation. A wrong decision leaves no stack trace. You build the record.

Write the exit conditions before you write the instructions. OpenAI's list is a good baseline: a final-output tool fires, the model returns a response with no tool calls, an error surfaces, or you hit the turn limit. Only one of those four is a success.

Why an agent is a distributed system with an unreliable actor inside

You have two processes, a network between the two, no shared clock, and a node that returns a different answer for identical input. That is a distributed system, and the unreliable node holds the decision.

The familiar catalogue arrives intact. A tool call can fail after the call changed state. A response can be lost after the work succeeded, and now you cannot tell whether a retry is safe.

Two workers can pick up the same item. A retry can land on a dependency that is already at the limit and multiply the load.

The prompt version and the tool version can also disagree, because you ship them on different schedules. Every one of those hazards is in how to design a distributed system, and none gets easier because the caller is a language model.

Where the analogy breaks

A crashed node is loud; a model failure is quiet, fluent and well formatted.

A dead process stops answering, and a breaker trips. A model that has misread the task keeps calling tools, returns schema-valid JSON, and writes a confident summary. Nothing in the transport layer notices, because the transport layer was never asked to check meaning. That is why error handling is not enough here, and why you move validation from did the call succeed into is the resulting state correct.

Latency behaves differently too. A model call sits anywhere from a few hundred milliseconds to a minute, and a five-step run compounds that spread, so the tail sets your concurrency budget rather than the mean. Cost is metered per token, which makes every retry policy a budget policy as well as a reliability policy.

State and memory: what survives a crash, what survives a context window

Keep three stores apart: the checkpoint that survives a crash, the memory that survives a conversation, and the context window that survives nothing.

LangGraph names the split cleanly. A checkpointer persists graph state per thread, which is what resumption, human-in-the-loop and fault tolerance need. A store persists application data across threads for preferences and facts (LangGraph persistence).

Temporal takes a different route to the same place: durable execution replays an event history, so a crashed worker resumes mid-workflow instead of restarting the run (Temporal, durable execution). LangGraph also exposes the failure that bites first in production: per-node retries, timeouts and error handlers, plus graceful drain and resume (LangGraph fault tolerance).

Anthropic is candid about why durability is not optional. Long-running agents hold state across many tool calls, and we can't just restart from the beginning because restarts are expensive and users feel them. The system therefore resumes from the last checkpoint and pairs model adaptability with deterministic retries and regular checkpoints (Anthropic, multi-agent research system).

The context window is the weakest of the three stores, and the evidence is not subtle. Lost in the Middle found accuracy peaks when the relevant passage sits at the start or end of a long input and degrades in the middle, even for models built for long context (Liu et al., 2023).

So what you put in the window is a ranking decision with a measurable failure mode, not a formatting decision. Anthropic treats context as a budget to spend deliberately, compacted and re-retrieved rather than accumulated (context engineering for agents). That budget argument is the same one behind LLM clusters becoming memory systems from the infrastructure side.

One rule covers most of this section: write the checkpoint before the effect, not after. If the next step needs a fact, put that fact in durable state, not in a message array you hope to replay. The transcript is a cache. Treat the checkpoint as the truth.

Tool contracts: the part of an agent you actually design

A tool is an API for a caller that cannot read your documentation, so the name, the description and the schema are one contract, and all three are load-bearing.

The Model Context Protocol makes the shape explicit. A tool is a name plus metadata plus an input schema, with structured content, an optional output schema and defined error handling (MCP specification, tools). MCP also states the safety expectation without hedging: for trust and safety, there SHOULD always be a human in the loop with the ability to deny tool invocations.

Anthropic's framing is the one I would put on a slide: tools are a new kind of software which reflects a contract between deterministic systems and non-deterministic agents (Writing effective tools for agents). That sentence explains why your usual API habits fail here. Your client is not reading the docs before the call. Your client is guessing, plausibly, from the name and the description.

Anthropic's earlier post on building effective agents reports a detail worth stealing: on its SWE-bench agent, the team spent more time optimizing tools than optimizing the overall prompt (Anthropic, Building effective agents). One fix came out of that work. The model kept mangling relative file paths after moving directories, so the tool was changed to accept absolute paths only.

What a tool schema must pin down

Pin down nine things, and the model stops guessing.

1. One verb per tool. cancel_order and refund_order are different tools with different blast radii, not a mode flag on update_order.
2. Absolute identifiers. No relative paths, no the current one, no implicit tenant from context the model cannot see.
3. Enums over free text for anything with a closed set of values, so an invalid choice fails at the schema instead of at the database.
4. Units in the field name. amount_cents, timeout_seconds, window_days. The ambiguity you leave is the bug you ship.
5. Explicit required vs optional, with a documented default for every optional field.
6. A success discriminant in the response, so an empty result and a failed call cannot look identical.
7. A structured error taxonomy with a retryable flag, so the model can choose between retrying and changing approach.
8. A response size cap, because a tool that dumps 200 KB of JSON consumes the context you needed for the decision.
9. A dry_run flag on anything irreversible. Cheap to add, and it turns a class of accidents into a preview.

Namespacing matters more as your tool count grows; Anthropic recommends grouping tools by service so the boundaries are obvious from the name. The other principles are unglamorous and effective: return meaningful context rather than raw payloads, optimize responses for token efficiency, and prompt-engineer the description as carefully as the system prompt.

Then add a governor, because the failure mode of a retry policy is a self-inflicted outage. Google's SRE book walks the arithmetic: a frontend sending 10,100 QPS at a backend that caps at 10,000 retries the rejected 100, and those retries add load to a service that is already at its limit (SRE Book, cascading failures). gRPC ships the fix as a token bucket — with maxTokens 10 and tokenRatio 0.1, failures drain the bucket, successes refill the bucket slowly, and retries pause once the count falls below half (gRPC retry guide). Exponential backoff with jitter spreads the herd while the bucket holds the line (AWS, backoff and jitter).

Idempotency closes the one window retries cannot. Stripe's model is the reference: the client sends a key, the server stores the first result, and a replay returns that result instead of doing the work twice (Stripe, idempotent requests). Only one failure window needs this — the operation succeeded and the connection broke — and that is precisely the window where an agent, which retries by design, will hurt you.

Derive the key from the logical intent, such as refund for order 4471, once, rather than minting a fresh key per attempt. An agent reintroduces every classic distributed-systems hazard at a higher rate, so the boring machinery earns its keep twice over.

Sandboxing and least privilege for tools that can do damage

Assume the model will be talked into misusing any tool within reach, then make the boundary a process boundary instead of a sentence in the prompt.

Claude Code starts read-only by default and asks before modifying anything. That is the safe shape and a terrible experience at scale. Anthropic's answer was OS-level isolation with two boundaries: filesystem isolation, so the agent can only touch specific directories, and network isolation, so the agent can only reach approved hosts (Anthropic, sandboxing).

You need both, and the post explains why in one line: without network isolation a compromised agent exfiltrates your SSH keys, and without filesystem isolation it escapes the box and gets network access anyway. The measurement is the interesting part: in Anthropic's internal usage, sandboxing cut permission prompts by 84%.

The isolation primitive is a design choice with a cost curve. Firecracker microVMs give VM-level isolation with five emulated devices, a small enough attack surface to reason about (Firecracker). Containers cost less per unit and share a kernel, so pair them with a hardened baseline like the restricted Pod Security Standard (Kubernetes PSS). Choose the boundary by blast radius: reading a public wiki does not need a microVM, and running model-authored shell does.

Credentials are the part teams forget. Give each tool a scope, not the agent's identity. Mint short-lived tokens per run, keep ambient cloud credentials out of the environment entirely, mount filesystems read-only unless a write is the point, and put an egress allowlist in front of anything that can make an outbound request. Permission modes are a real control surface, not documentation (Claude Code permissions).

Human approval gates: approve consequences, not steps

Gate on what cannot be undone or paid for, and let everything else run.

OpenAI's guardrail taxonomy gives you the scoring rubric. Rate every tool low, medium or high on read-only versus write, reversibility, required permissions and financial impact, then let the rating choose the control — an extra check, a pause, or a person (OpenAI, practical guide). The same guide names the two triggers that warrant a human: exceeding a failure threshold, and any high-risk action, defined as sensitive, irreversible or high stakes.

Make approval a durable pause, not a blocked thread. Checkpoint the run, emit the pending decision as state, and resume when a decision arrives. Give the approval a timeout that denies by default, because an approval queue nobody watches is an unattended authority. Frameworks expose this as an interrupt that suspends the graph and resumes with a value (LangGraph interrupts), and you want that shape regardless of framework.

The failure mode of approval design is approval fatigue. Anthropic's 84% number exists because per-action prompts train people to click approve without reading. Gate the few actions that move money, send external messages, delete data or change permissions.

Let reads and cache lookups run. A gate that fires ten times an hour is not a control; a gate that fires ten times a month is.

Observability: traces are the only ground truth you get

Log the decision, not just the request — one span per model call and per tool call, carrying inputs, outputs, retries, latency, tokens and the state change.

Anthropic describes the exact symptom you will meet. Users reported the agent not finding obvious information, and the team could not tell whether the cause was a bad query, a bad source, or a failing tool. Full production tracing turned an argument into a diagnosis (multi-agent research system). That is the value proposition, and the value is not dashboards.

Use the OpenTelemetry generative AI conventions for span attributes so your traces stay portable across vendors and backends (OTel GenAI semantic conventions). Then record what a replay needs: run id, step index, prompt version, model version, tool name, a hash of arguments, retry count, the approval decision and who approved, token counts, and the state delta. Skip the whole prompt body if it carries customer text. A hash plus a pointer into redacted storage is enough for correlation and safer in retention.

Two habits make traces useful. Emit the span before the tool call and close the span after the observation, so a hung call shows up as an open span rather than a silent gap. And correlate the run to the alert you will eventually get: the trace is the first artefact in the incident triage pipeline, not a debugging luxury.

Evaluation harnesses for a system that never runs twice

Stop asserting on text; assert on the state of the world after the run, and grade the transcript only where the state is ambiguous.

τ-bench is the cleanest published version of this. The benchmark compares final database state to an annotated goal state instead of diffing the agent's prose, and introduces pass^k to measure whether the agent succeeds on all of k attempts (Yao et al., τ-bench). The results are the argument for the metric: a state-of-the-art function-calling agent succeeded on under 50% of tasks, with pass^8 under 25% in retail. A single-run success rate flatters a system that has to work on Tuesday.

Anthropic's eval vocabulary is worth adopting verbatim, because the words force precision. A task is one test with success criteria, an attempt is a trial, a transcript is the full record of a trial, and the outcome is the final state in the environment. Anthropic's example is on point: a booking agent can say your flight has been booked while the reservation table disagrees (Demystifying evals for AI agents).

The same post reports a stronger model solving a τ²-bench task by finding a policy loophole. The model failed the eval as written while producing a better answer for the user. Your graders need judgement, and the answer is more trials plus human review of disagreements, not a stricter string match.

A harness that earns its keep has five properties:

• You seed the environment, so every trial starts from the same world.
• You can inject faults: a tool that returns 500, a tool that returns empty, a tool that is slow.
• Your tasks come from real traffic, not from imagination.
• You rerun on every prompt and model change, not quarterly.
• You report cost per successful task beside the pass rate. A design that buys three points of accuracy for four times the tokens is a product decision someone should make on purpose.

Report distributions, not scores. If a change moves the pass rate from 71% to 74% across ten trials, you have measured noise. Eight trials per task is a common starting point precisely because eight exposes the variance that one run hides.

Hard ceilings: steps, tokens, time and money

Every run gets four ceilings enforced by your code: maximum steps, maximum tokens, maximum wall clock, maximum spend. A prompt is not an enforcement mechanism.

The economics explain the need. Anthropic measured agents at roughly 4× the tokens of a chat interaction and multi-agent systems at roughly 15×, and found that token usage alone explained 80% of the performance variance on one browsing benchmark (multi-agent research system). You buy capability with tokens, so the budget is a design parameter, not an accounting detail.

Enforce the ceilings in the orchestrator, where the model cannot argue. A step limit catches loops. A token limit catches runaway context. A wall-clock limit catches a hung tool.

A spend limit catches the combination that passes all three, and that is the only one your finance team will read. Add a circuit breaker per dependency so a failing tool gets cut out instead of retried forever (Fowler, CircuitBreaker), and a per-tenant bulkhead so one customer's agent cannot consume the shared quota. When a ceiling trips, say so: return partial work, name the ceiling that fired, and hand control back with the trace attached. Silent truncation reads as incompetence; an explicit stop reads as a system with limits, which is what you built.

The failure modes that actually show up in production

Loops, silent tool failure, injected instructions in retrieved content, and duplicated side effects account for most production incidents, and none of them has a prompt fix.

Loops and thrash

A loop is not a bug in the model; a loop is a missing progress check in your orchestrator.

The common shape is a repeated tuple: same tool, same arguments, same error, four times over. A subtler shape is two tools alternating — read the file, list the directory, read the file — each step plausible, none adding information. You can detect both without asking the model anything.

Hash the tool name and arguments per step. Abort when a tuple repeats more than twice. Require a state delta between steps, and treat no new information as a failed step rather than a successful one. Then escalate with the transcript, because a loop that trips a limit is a design signal about the tool description, not just a run to kill.

Silent tool failure

A tool that returns 200 with an empty body is more dangerous than one that returns 500.

An error forces a decision. An empty result looks like an answer, so the agent summarises nothing with confidence and moves on. Fix this in the tool: return a success discriminant, distinguish no matching records from the query failed, and validate the payload before the model sees it. Then tell the agent when a tool is failing, because Anthropic reports that letting the model know about a failing tool and letting it adapt works surprisingly well (multi-agent research system).

Track tool error rate as a service-level indicator, with empty-result rate beside it. A tool whose empty rate jumps from 2% to 40% is broken, and no amount of model quality compensates.

Prompt injection through retrieved content

Retrieved text is data, and the model has no reliable way to know that.

This is the failure mode with the best documentation and the worst adoption. OWASP ranks prompt injection as the first LLM risk and states plainly that retrieval and fine-tuning do not fully mitigate prompt injection (OWASP LLM01). Greshake and coauthors showed in 2023 that an attacker can inject instructions into content the model will later retrieve, blurring the line between data and instruction (indirect prompt injection). Morris II showed the same trick propagating between assistants, worm-style, through a RAG pipeline (AI worm).

The incidents are not hypothetical. Invariant Labs demonstrated a malicious public GitHub issue hijacking a coding agent into leaking private repositories through the official GitHub MCP server (GitHub MCP exploited). PromptArmor showed Slack AI exfiltrating private-channel content after ingesting a poisoned public message (Slack AI exfiltration). Microsoft patched CVE-2025-32711, an AI command injection in Microsoft 365 Copilot rated CVSS 9.3 with no user interaction (MSRC advisory).

Simon Willison's framing is the one to design against: the dangerous combination is private data access, exposure to untrusted content, and an outbound channel, because together the three form an exfiltration path (lethal trifecta). You cannot patch the model out of that. You can refuse to build the trifecta into one loop.

Four controls carry most of the weight:

• Delimit untrusted content and mark the boundaries as data in the prompt, knowing that marking is a mitigation and not a boundary (OWASP prevention cheat sheet).
• Split capabilities so the loop that reads private data cannot also send externally.
• Allowlist egress, both for the sandbox and for the outbound tools.
• Put a human on anything that leaves the building, because the last hop is the one that needs consent.

Tool output must never be able to add tools or change policy. If retrieving a document can widen the agent's authority, your retrieval layer is the control plane.

Non-deterministic side effects

The model is non-deterministic; your side effects cannot be.

A retried step re-sends the email. A resumed run charges the card twice. A duplicated message fans out to a webhook that is not idempotent.

The structural answer is the same as for any unreliable producer: an idempotency key per logical action, an effect ledger written in the same transaction as the effect, and an outbox for anything that leaves the system. Reconcile on a schedule, because reconciliation catches the case where the effect landed and the record did not.

Commitments count as side effects, and this is where the legal risk lives. Air Canada argued in a Canadian tribunal that the website chatbot was a separate legal entity that is responsible for its own actions. The adjudicator called that submission remarkable, held the airline responsible for what the site says, and ordered CA$812 in compensation on a CA$1,630.36 fare (CBC News). Treat outbound text as a write with consequences, because a promise a customer can rely on is exactly that.

What this post is not — and what to read instead

This is not a framework tutorial, a prompt cookbook, or a model comparison. Architecture only: the structure you own regardless of which vendor answers the call.

For the neighbouring ground, three siblings. Must-have LLM skills and agents for engineering managers covers the management side — what to ask for in a design review and how to staff this work. Designing production-ready RAG systems goes deep on the retrieval layer, where injected content enters and where chunking, ranking and freshness decisions get made. And the system design practical guide is the foundations: capacity, SLOs, idempotency and failure budgets, minus the model.