A chatbot that gets something wrong simply produces an incorrect response. You can read it, reject it and try again. The consequences are generally limited to the output. An agent that makes a mistake can leave an entire system in the wrong state. It could make the same payment twice, alter a record it was never supposed to touch, or stop midway through a workflow without leaving a clear trail of what was completed. The output is no longer the main artifact. The resulting state of the world is.
The architecture has changed accordingly. Generative AI followed a relatively simple pattern: prompt, model, output. Agentic AI involves a much longer chain: intent, reasoning, authorization, tool use, external action, state change, verification and recovery. The first architecture primarily generates information. The second can generate consequences. Most of the reliability challenges that matter in production therefore sit in the parts of the pipeline that extend beyond the model itself.
This is the shift I believe the industry has yet to fully appreciate. We continue to discuss model accuracy as though it were the complete measure of reliability, even as probabilistic systems are increasingly connected to tools, databases, payment systems and other agents. Model reliability remains important, but once AI becomes part of stateful, multi-step processes, it is both essential and insufficient. An agent's reliability is no longer determined solely by its model. It depends on the runtime architecture built around that model.
What the benchmarks already tell us
There is already evidence supporting this argument. Sierra's t-bench, among the relatively few benchmarks designed to test agents in stateful environments, introduced a metric known as pass^k. It measures the probability that an agent will successfully complete the same task across all k attempts, rather than succeeding just once. The findings should concern anyone putting agents into production. Even the strongest function-calling agents tested completed less than half of the tasks successfully, while consistency across eight attempts dropped below 25 percent in the retail category.
That distinction matters. The system, task and tools can all remain identical, yet repeated execution can still produce different outcomes. A successful demonstration establishes that an agent is capable of doing something. Production deployment requires it to do that thing consistently, along with a runtime capable of detecting when consistency breaks down.
The same pattern appears in multi-agent systems. A University of California, Berkeley team examined more than 1,600 execution traces across seven widely used frameworks and developed a systematic taxonomy of their failure modes. They identified 14 distinct failure modes spanning specification and system design, coordination between agents and task verification. Many failures could not be explained simply by shortcomings in model reasoning. Instead, they emerged from architecture, coordination and verification issues, including unclear responsibilities, lost context and agents proceeding without confirming whether previous actions had actually succeeded. The researchers also showed that improving system design could produce performance gains without depending entirely on better models, reinforcing the central argument here.
The problems are older than the models
Here is the uncomfortable reality for the AI industry: many of the reliability challenges being described as new are actually longstanding problems. They are distributed-systems problems appearing in a new environment. Once a model starts coordinating calls to external tools and stateful services, its runtime inherits familiar distributed-systems failure modes whether its designers planned for them or not. Partial failures, retries, timeouts, concurrency, dependency failures and ambiguous delivery all become relevant.
Consider a simple example. An agent sends a payment request to an external API. The API successfully executes the transfer, but the response times out before the agent receives confirmation. The system is now genuinely uncertain. Did the payment fail, or did the transfer succeed while only the acknowledgment was lost? Should the agent attempt the payment again? If the operation is not idempotent, a retry could result in the customer being charged twice. There is no hallucination involved here, and more sophisticated reasoning cannot solve the underlying problem. If the payment happened and the confirmation disappeared, intelligence cannot recover information that was never received. The solution lies in architecture: idempotency keys, durable state and reconciliation with the authoritative system of record.
Engineers working on hyperscale systems developed approaches to these problems long before AI agents existed. Amazon's Builders' Library explains how retries without idempotency can duplicate side effects and how client request tokens can ensure repeated calls resolve to the same outcome as the original request. Related guidance on timeouts, retries and backoff with jitter shows how poorly designed retry mechanisms can turn a minor failure into a synchronized system-wide surge. None of these principles were created specifically for AI, but they now apply directly because an agent is, in effect, a distributed-systems client that can reason.
The probabilistic-deterministic boundary
What is genuinely different is the nature of the component making decisions. Language models are probabilistic by design. Yet many of the guarantees they increasingly interact with cannot be probabilistic. A payment cannot be "probably" made exactly once. An authorization boundary cannot be "usually" enforced. A ledger cannot be "mostly" consistent. An irreversible action affecting a customer cannot be "approximately" committed.
That makes one of the central responsibilities of an agentic runtime the translation of probabilistic reasoning into controlled, deterministic execution. The model can suggest what ought to happen. The runtime must determine whether the action is authorized, whether the underlying state remains current, whether the action has already taken place, whether retrying is safe, whether the action actually succeeded and whether the resulting state satisfies the business rules that cannot be compromised. The model proposes; the runtime commits.
The concept of commitment deserves particular attention because it exposes one of the largest gaps in today's agent architectures. Production systems must distinguish between proposing an action, authorizing it, attempting it, committing it, confirming the resulting state and recovering or compensating when something goes wrong. A model producing the instruction "transfer $500" does not mean $500 has actually left an account. A tool returning an error does not necessarily establish that the transfer failed. Likewise, a successful API response does not necessarily prove that the broader business transaction was completed. Database systems addressed similar concerns through concepts such as atomicity and rollback, although the comparison is imperfect. Many external actions cannot be reversed, which makes explicit commit points and compensating actions even more important for agents.
I refer to this as the agentic commit boundary: the moment when a proposed action moves from the realm of reasoning into a real-world consequence. The safeguards required before crossing that boundary should correspond to the potential impact. Reading a publicly available document may require very little control. Deleting production data should require validation, staging and a recoverable checkpoint. Transferring money should require explicit authorization, idempotency and verification against the ledger. This leads to a more useful definition of autonomy: it is not simply how much an agent can decide, but where it is allowed to change state without requiring additional assurance.
Where reliability actually lives
Once agent execution is viewed as a series of controlled state transitions rather than a model simply producing an answer, the location of reliability becomes clearer. It spans six layers.
Model reliability: Can the model reason sufficiently well for the task?
Context reliability: Did the system retrieve the correct information and prevent stale or contradictory context from influencing the decision?
Tool reliability: Did the correct API execute with the intended parameters and semantics?
State reliability: Does the system know what has already happened, what remains pending and whether an action actually took effect?
Workflow reliability: Can a multi-step process withstand timeouts, partial failures and substitutions without corrupting the final result?
Governance reliability: Was the action authorized, can the behavior be reconstructed afterward and can a human intervene when the consequences require it?
The importance of these categories lies not in the categories themselves, but in the fact that agent reliability is compositional and end-to-end. It emerges from the entire execution path and cannot be predicted from model accuracy alone. Every consequential step creates another opportunity for context, state, authorization or execution to diverge from expectations. That is exactly the type of behavior exposed by pass^k. A highly capable frontier model operating inside a runtime that cannot determine whether a payment actually succeeded is still an unreliable agent.
Viewed sequentially, the stack becomes a practical architecture: intelligence — can the model identify an appropriate action? Authority — is this execution allowed to perform it? State — does the system know what has already occurred? Commit — can the proposed action safely become external state? Verification — did the real-world state change as intended? Recovery — what happens if it did not, or if the outcome cannot be determined? Provenance — can the complete sequence be reconstructed afterward? If one of these stages is missing, production systems will eventually expose the weakness.
That is why orchestration is increasingly becoming the real reliability layer, and why it needs to mean more than simply connecting multiple agents. Production systems require an agentic control plane: a layer responsible for identity, authorization, context assembly, workflow state, retries, idempotency, fallback, verification, escalation, auditing and recovery. Its purpose is to enforce invariants around probabilistic execution rather than merely coordinate components.
Even Anthropic's engineering guidance distinguishes predefined workflows from agents that independently direct their tool use, and recommends beginning with the simplest solution possible, including the possibility that an agentic system may not be necessary. Every additional control-plane layer introduces complexity, latency, cost and its own potential failure modes. The objective should therefore not be the most sophisticated agent architecture, but the smallest architecture capable of safely handling the consequences being delegated to it.
Fallback also deserves more scrutiny. The assumption that Model A can simply be replaced by Model B when it fails treats models as interchangeable components. They are not. Two models may expose compatible APIs while behaving differently in instruction interpretation, tool selection, structured output, ambiguity resolution and refusals. API compatibility does not establish behavioral compatibility, and syntactic interchangeability does not guarantee semantic interchangeability. Fallback is therefore not simply a redundancy mechanism. It is a semantic compatibility problem. The runtime needs to determine whether a replacement model preserves the behavioral assumptions under which the workflow was originally validated. Conventional infrastructure failover generally assumes a stronger level of behavioral equivalence than model substitution can safely provide.
Identity is similarly a delegation problem. Conventional access management asks who a principal is and which resources that principal can access. Agentic systems introduce a more difficult set of questions: who granted the authority, for what purpose, over which resources, subject to which restrictions, for how long and with what ability to delegate that authority further?
An agent should not automatically inherit all the permissions of the person who initiated it, as can happen when it operates through a human user's session. Its authority should instead be constrained by the task, resources and duration, while remaining revocable and auditable. OWASP's AI agent security guidance similarly recommends least-privilege tool access, step-up authentication for high-impact actions, idempotency for critical operations and fail-closed behavior when authorization or audit mechanisms fail. The underlying principle is straightforward but difficult to implement: an agent being technically capable of calling a hundred tools does not mean that a particular execution should be authorized to use any of them. Capability must never be mistaken for authority.
Observability also takes on a different form. Traditional monitoring focuses on whether a service is available and whether requests are succeeding. Agentic systems require decision provenance: not a reconstruction of every internal thought of the model, but a record of what the system knew, what decision it made, what authority it possessed, what actions it attempted and what actually changed. The OpenTelemetry generative AI semantic conventions are beginning to standardize this area by defining trace structures for agent invocations, model calls and tool executions, making it possible to reconstruct an execution path after a failure. That is a narrower objective than explaining a model's reasoning and, importantly, one that is much more achievable.
Evaluating systems, not models
This is also where benchmarking needs to evolve. A benchmark can provide meaningful information about a model, but it cannot establish whether an entire workflow recovers properly after a partial failure, prevents duplicate actions, respects authorization limits or escalates when uncertainty becomes unacceptable. Those are system-level properties that must be tested against the system itself, with pass^k-style consistency measured against actual state rather than static prompts on a leaderboard.
Governance frameworks are already moving in this direction. NIST's Generative AI Profile structures its recommendations around areas such as governance, content provenance, pre-deployment testing and incident disclosure. These are process and system controls rather than simple model metrics. Model evaluation measures capability; system evaluation measures whether that capability can be translated into dependable execution. Increasingly, the latter will determine whether agents are suitable for production.
Generative AI made reasoning probabilistic. Agentic AI places that probabilistic reasoning in control of consequential actions, creating a fundamentally different engineering challenge. The first phase of the AI industry focused on making the models more intelligent, and that effort delivered significant progress. The next frontier lies in the boundary around those models: the systems that decide when intelligence can act, what it can change, how those changes are verified and what happens when reality fails to match expectations.
The defining question for agentic AI will not simply be whether a model can reason its way to the correct action. It will be whether the runtime can ensure that the intended action takes place under the correct authority, against the correct state, without unsafe duplication, while also providing a path to recovery when any assumption breaks down. The model provides the intelligence. The runtime determines what that intelligence is allowed to make real.
Dr. Aditya Vikram Kashyap is an AI governance researcher and practitioner whose work focuses on institutional accountability, agentic AI, model risk and AI governance in regulated institutions. His broader work examines emerging technology, financial services and systemic technology risk. The views expressed are his own.
