State of Agent Infrastructure: The Annual Stack Report

Table of Contents

• Why an Annual Stack Report Now
• The Stack, Bottom to Top
• Layer 1: Inference and Model Routing
• Layer 2: The Agent Runtime
• Layer 3: Orchestration and Durable Execution
• Layer 4: Memory, State, and Retrieval
• Layer 5: Tools, Protocols, and the Gateway
• Layer 6: Observability and Eval
• What Consolidated This Year
• What Is Still a Mess
• The Economics Underneath the Stack
• How GaaS Operators Should Read This Map
• Insights Most People Overlook
• References

Why an Annual Stack Report Now

A year ago, "agent infrastructure" mostly meant a prompt loop wrapped around an API call, plus whatever glue code a team wrote on a Tuesday. The phrase covered everything and therefore meant nothing. That has changed. There are now distinct vendors, distinct categories, and, the real tell, distinct budget lines for pieces of the stack that did not have names eighteen months ago.

The reason to take inventory annually is that this layer moves faster than the model layer it sits on. Frontier models improve on a roughly predictable cadence and a handful of labs ship them. Infrastructure, by contrast, is being invented in public by hundreds of teams reacting to whatever broke last quarter. A retry-and-fallback pattern that was bespoke code in one company becomes a product category in another. The map is worth redrawing because last year's map is already wrong in three places.

For anyone running Agentic AI-as-a-Service, this is not academic. When you sell per-task or per-outcome pricing, every layer of the stack is a line in your cost of goods sold. A bad orchestration choice doesn't just hurt reliability, it shows up directly in gross margin. So this report is organized the way an operator should think about it: what each layer does, what it costs, and where it's likely to change under you.

The Stack, Bottom to Top

The cleanest way to read the agent stack is as six layers, from the silicon-adjacent bottom to the operator-facing top. Most vendors claim to span more than one. Almost none do all six well.

Layer 1: Inference and Model Routing

At the base sits raw inference: the GPUs, the inference providers, and increasingly a routing layer that decides which model handles which call. The story here this year is the divergence between "frontier for hard steps" and "cheap, fast model for everything else." Agents are token-hungry in a way that chat never was, a single autonomous task can fan out into dozens of model calls, so naive routing where every step hits the most capable model is how GaaS companies quietly go broke.

Model routing matured from a clever trick into expected infrastructure. The premise is simple: classify the difficulty of a step, then send it to the cheapest model that can handle it. The execution is not simple, because misrouting a hard step to a weak model produces a failure three steps later that's expensive to trace. The inference providers themselves are now competing explicitly for agent workloads, optimizing for throughput on long, bursty, multi-call sessions rather than single-shot latency. Andreessen Horowitz's running analysis of the emerging LLM infrastructure stack captures how quickly this base layer has stratified into specialized providers.

Layer 2: The Agent Runtime

This is the category that genuinely earned a name this year. The "agent runtime" is the thing that actually executes an agent's loop: it manages the turn-by-turn cycle of model call, tool call, observation, and decision, and it holds the agent's working context while that loop runs. Think of it as the JVM for agents, the layer that turns a definition into a running, stateful process.

Why this matters: for a long time the runtime was buried inside the orchestration framework, indistinguishable from it. Teams are now pulling it out because the concerns are different. A framework helps you author an agent; a runtime runs it in production, with all the messy requirements that implies, pausing mid-task, surviving a process restart, resuming exactly where it left off. The runtime is where sandboxing, resource limits, and per-step isolation get enforced. If you've ever had an agent spawn a runaway subprocess or quietly burn $40 in tokens on a loop that never terminates, you've felt the absence of a real runtime.

Layer 3: Orchestration and Durable Execution

Above the runtime sits orchestration: how multiple steps, and often multiple agents, are coordinated into a workflow. This is the most crowded and most contested part of the stack, and it's where the framework wars play out.

The important shift this year was the arrival of durable execution as a first-class requirement rather than a nice-to-have. Agents that run for minutes or hours, research tasks, multi-stage data pipelines, anything with human-in-the-loop checkpoints, cannot live in memory on a single machine. When that machine reboots, the task must survive. Durable execution engines, borrowed conceptually from the workflow-orchestration world that predates LLMs, treat each step as a checkpointed, replayable event. The agent's progress becomes a log you can resume, audit, and retry. This is the single biggest reliability upgrade most GaaS teams can make, and it's still underadopted because it requires rethinking the agent as a state machine rather than a script.

Supervisor architectures, one coordinating agent delegating to specialized workers, settled into the default pattern for complex multi-agent work, largely because flat "swarms" of peer agents proved hard to debug and harder to bound in cost.

Layer 4: Memory, State, and Retrieval

Memory is where the stack is least settled and the marketing is loudest. The honest summary: there is no single memory architecture, there are tradeoffs, and most teams are over-engineering this layer.

The components are real, though. Short-term working memory lives in the context window and is governed by token budgets. Long-term memory persists across sessions and usually leans on a vector store, a key-value store, or both. Retrieval pulls the right slice of that memory back into context at the right moment. The interesting development this year was the quiet reconsideration of whether a dedicated vector database is even necessary for many agent workloads, for smaller corpora, in-process retrieval or a Postgres extension often beats a separate vector service on both cost and operational simplicity. The "context-window economy", deciding what an agent remembers, forgets, and re-fetches, became a genuine engineering discipline, because tokens carried into every step are tokens you pay for on every step.

Layer 5: Tools, Protocols, and the Gateway

Agents are only as useful as the tools they can call, and this year the tool layer got its first real standards. The Model Context Protocol emerged as the leading candidate for how agents discover and call external tools and data sources, giving the industry something it badly lacked: a common way to define a tool once and expose it to many agents. Anthropic's Model Context Protocol documentation lays out the spec that a growing share of the ecosystem now builds against. Alongside it, agent-to-agent protocols started defining how separate agents, possibly from different vendors, hand work to each other.

Sitting in front of all of this is the agent gateway: the policy, routing, rate-limiting, and auth layer that every tool call and model call passes through. In enterprise deployments this is non-negotiable. It's where you enforce which tools an agent may touch, throttle runaway behavior, attach identity, and log everything for audit. The gateway is to agents what the API gateway was to microservices, and it's maturing on the same trajectory.

Layer 6: Observability and Eval

You cannot operate what you cannot see, and agents are uniquely hard to see into. A single task produces a branching tree of model calls, tool invocations, retries, and decisions. The observability layer, tracing, logging, token accounting, and increasingly automated evaluation, moved from afterthought to table stakes this year.

The distinctive challenge is that agent observability has to capture reasoning, not just requests. Knowing that a call failed is easy; knowing why the agent decided to make that call, and whether the decision was correct, requires capturing intermediate state most logging stacks were never designed to hold. Evaluation infrastructure, running agents against simulated environments and scoring outcomes before they reach production, is the natural companion, and it's where the more sophisticated GaaS operators now spend real engineering time.

What Consolidated This Year

Three things genuinely firmed up.

First, the runtime separated from the framework. The market now broadly agrees these are different products with different buyers, even if individual vendors bundle them.

Second, durable execution went mainstream as the answer to long-running reliability. The debate moved from "do we need it" to "build or buy," which is how you know a category has crossed the chasm.

Third, MCP became the gravitational center of the tool-definition conversation. It is not the only protocol and it may not win permanently, but it gave the industry a shared vocabulary, and shared vocabulary is how fragmentation eventually ends. McKinsey's analysis of the agentic AI shift for enterprises frames why standardization at this layer is the precondition for enterprise-scale adoption.

What Is Still a Mess

Plenty did not consolidate.

Memory remains a Tower of Babel, every framework has its own abstraction, none interoperate, and "memory" means five different things depending on the speaker. Agent-to-agent interoperability is real in demos and shaky in production; who controls the standards is an open and somewhat political question. Identity and auth for agents, how you give an autonomous process scoped, revocable, auditable access to systems built for humans, is the most under-built critical layer in the entire stack. And the GPU-supply constraint quietly shapes everything above it: routing, caching, and self-hosting decisions are often driven less by architecture preference than by what compute you can actually get.

The Economics Underneath the Stack

Here is the part most stack reports skip. For a GaaS business, the stack is a cost structure, not just an architecture diagram.

Every layer adds cost and adds reliability, and they trade against each other. Skipping durable execution saves infrastructure spend until a multi-hour task dies at minute fifty-eight and you eat the full token cost with nothing to show. Routing every call to the frontier model maximizes quality and destroys margin. Caching aggressively cuts cost but risks serving stale results in a workflow where freshness matters. The operators who survive on per-outcome pricing are the ones who treat each layer as a dial to be tuned against a unit-economics target, not a checkbox to be ticked.

The non-obvious cost driver is context. Tokens carried into every step compound: a verbose system prompt or a bloated memory retrieval doesn't cost once, it costs on every model call for the entire length of the task. Managing the token budget at runtime is, for many agent workloads, the highest-leverage cost optimization available, higher than model choice, because it multiplies across every call.

How GaaS Operators Should Read This Map

If you build and sell agents, the practical takeaways are short.

Invest in durable execution before you invest in more agents, reliability is the product. Treat the runtime as a deliberate choice, not a byproduct of your framework. Build observability in from the start, because retrofitting it onto a live agent fleet is miserable. Standardize your tool definitions around MCP now while it's cheap, rather than after you've accumulated a hundred bespoke integrations. And model your stack as a cost structure: every layer is a line in your COGS, and on per-outcome pricing, the infrastructure is the margin.

The model layer gets the headlines. The infrastructure layer gets the business.

Insights Most People Overlook

• The runtime, not the framework, is the real lock-in. Everyone worries about framework lock-in, LangChain versus the alternatives, but frameworks are mostly authoring conveniences you can swap. The runtime holds your live state, your checkpoints, and your execution semantics. Migrating that mid-flight is the genuinely hard switch, and almost nobody evaluates runtimes with that in mind.

• Durable execution quietly changes your pricing model. Once tasks are checkpointed and resumable, "per-outcome" pricing becomes defensible because you can actually guarantee completion. Without durability, per-outcome pricing is a bet you'll lose on long tasks. The infrastructure choice and the business model are the same decision.

• The vector database may be a liability, not an asset, for many agents. The reflex to reach for a dedicated vector store is a holdover from RAG-era thinking. For the modest corpora most vertical agents actually touch, a separate vector service adds operational surface, cost, and latency for retrieval quality you could get from Postgres. Question this layer before adopting it.

• Identity is the layer that will cause the first big agent breach. The stack has rushed to give agents power, tool access, system credentials, autonomy, far faster than it has built scoped, revocable, auditable identity for them. The gap between what agents can do and how well we can constrain them is the most under-priced risk in the whole report.

• Observability that only captures requests is worthless for agents. If your tracing shows you the calls but not the reasoning behind them, you can see that an agent went wrong but never why. The teams treating reasoning capture as a first-class telemetry requirement are the ones who can actually debug production incidents instead of guessing.

References

• Emerging Architectures for LLM Applications, Andreessen Horowitz
• Model Context Protocol, Official Documentation
• The State of AI: Agentic AI for the Enterprise, McKinsey & Company