An AI agent is not just an LLM with tool definitions. A production agent is a runtime system that assembles context, chooses actions, interacts with external services, preserves state and proves what happened.
The architecture should make model-driven choice useful without allowing probabilistic reasoning to become the only control mechanism.
Start with the Augmented Model
The smallest useful unit is a model with bounded augmentations:
model + instructions + retrieval + tools + memory
That unit can answer, classify and call a tool. It becomes an agent when the model dynamically controls parts of the process: it decides which step comes next, which tool to use or whether the current evidence is sufficient.
Anthropic’s useful distinction is between workflows, where code defines the path, and agents, where the model directs its process and tool use. A good architecture supports both rather than forcing every task into agent autonomy.
The Eight Runtime Components
1. Request and identity boundary
Before reasoning begins, resolve the user, tenant, task, permissions and correlation ID. These are system facts, not prompt suggestions. They determine which data and tools may enter the run.
2. Context assembler
The context assembler creates the model’s current working view. It can combine instructions, retrieved documents, graph facts, recent messages, tool schemas and a concise state snapshot.
Context should be selected, not accumulated. Dumping every available record into a window increases cost and makes authority ambiguous. Each item should have a role: instruction, evidence, state, example or constraint.
3. Model operator
The model proposes interpretations, plans, arguments or tool calls. It should receive typed expectations whenever possible and return a candidate that the runtime can validate.
The model is replaceable when the contract is owned by the system rather than encoded only in model-specific prompt habits.
4. Orchestrator
The orchestrator owns the run lifecycle. It invokes the model, routes steps, enforces budgets, handles retries and knows when execution has finished.
For predictable tasks it follows a coded workflow. For uncertain tasks it may give the model a bounded choice among permitted transitions. This is where “agentic” becomes a degree of control rather than a binary product label.
The strategic implications are developed in AI Coordinator Architecture and Process Sovereignty .
5. Memory subsystem
Memory makes prior information available again. Short-term memory may be a thread checkpoint; long-term memory may store preferences, prior solutions or summarized experience across threads.
Memory can be incomplete, stale or selectively retrieved. It should therefore inform decisions without automatically becoming authoritative truth. Agent Memory Architecture develops the working, episodic, semantic and procedural layers; Stateful AI Agents defines the boundary between memory and authoritative state.
6. State and graph layer
State represents current facts about the run and domain. A graph adds addressable relations: which claim came from which source, which task depends on another, which agent may use a tool, or which asset version was approved.
This layer gives the agent a world it can query instead of asking the model to reconstruct the world from prose on every turn.
7. Tool gateway
The tool gateway is a policy boundary. It validates arguments, injects credentials outside the prompt, applies timeouts and records side effects. Tools that change data should support idempotency keys or equivalent duplicate protection where possible.
Do not expose a raw administrative API merely because the model can format its parameters. Offer narrow operations that express legitimate business actions.
8. Validation and observability
Every run should produce more than a final message. Record model and tool versions, selected sources, tool calls, state transitions, costs, latency and evaluation outcomes.
Validation happens at several levels:
- schema and type checks;
- permission and policy checks;
- evidence and citation checks;
- task-specific quality evaluation; and
- human approval before high-impact actions.
A Reference Agent Loop
load authoritative state
→ assemble bounded context
→ ask model for next permitted action
→ validate proposed action
→ execute tool or retrieve evidence
→ record observation
→ update run state
→ stop, continue or request approval
The word “permitted” is essential. The model does not invent the set of legal transitions. The production architecture supplies it.
Where Graphs Fit
Graphs are valuable when relations drive the decision. A customer-support agent may need to connect an account, contract, entitlement, incident and approved remedy. A research agent may connect claims, sources, authors and contradictions. A creative agent may connect characters, objects, locations, events and knowledge states.
The graph is not the agent’s chain of thought. It is an inspectable domain representation. Read more in Knowledge Graphs as the Agent Control Plane .
Choose the Minimum Necessary Autonomy
Use a deterministic function when the rule is known. Use a workflow when the sequence is known but some steps benefit from model judgment. Use an agent when the next useful step genuinely depends on observations that cannot be enumerated economically in advance.
This keeps the system understandable and controls cost, latency and error propagation. The resulting agent may be less theatrical than an “autonomous employee,” but it is more useful in production: bounded, observable and accountable.