How to Build a Multi-Model Agent Without Getting Locked Into One Vendor

Table of Contents

• Why Lock-In Is Different for Agents Than for Apps
• The Anatomy of a Portable Agent
• The Model Abstraction Layer: What It Must Normalize
• Tool-Calling Is Where Portability Breaks
• Routing: Choosing the Right Model Per Task
• The Evaluation Harness You Have to Own
• Build vs. Buy: Gateways, Frameworks, and Rolling Your Own
• The Economics of Portability
• Insights Most People Overlook
• References

Why Lock-In Is Different for Agents Than for Apps

When you build a normal app on top of a language model, the coupling is shallow. You send a prompt, you get text back, you move on. Swapping providers means changing an API client and re-tuning a handful of prompts. Annoying, but a week of work.

Agents are different because the model is woven into a control loop, not bolted onto the edge. An agent decides which tool to call, interprets the result, decides whether to call another, and eventually stops. Every one of those decisions depends on quirks of the specific model you trained the system around: how it formats tool calls, how aggressively it chains steps, how it behaves when a tool returns an error, how it interprets a system prompt that says "do not make up data." You don't just depend on the model's raw quality. You depend on its behavior inside a loop you designed around it.

That's the trap. By the time an agent is working in production, the lock-in isn't in the API call, it's distributed across your prompts, your tool schemas, your retry logic, your stop conditions, and the thousand small accommodations you made to get one specific model to behave. This is the same dynamic that makes the broader GaaS infrastructure cost stack so sticky: the value accrues in the orchestration glue, and the glue is where you get trapped.

I've watched teams discover this the hard way. A model provider raises prices or deprecates a snapshot, the team says "fine, we'll switch," and then realizes their agent's reliability quietly craters on the new model because none of the behavioral assumptions hold. Portability isn't a feature you add at the end. It's an architecture decision you make at the start, or pay 5x to retrofit.

The Anatomy of a Portable Agent

A portable agent separates four concerns that lock-in usually fuses together:

1. The model interface, a normalized contract for sending messages and receiving responses, independent of any provider's SDK.
2. The tool layer, tool definitions and execution that don't assume a particular provider's function-calling format.
3. The orchestration logic, the loop, the stop conditions, the memory, the state. This should be model-agnostic by design.
4. The configuration plane, prompts, routing rules, model selections, and parameters that live in config or a registry, never hardcoded into the loop.

When these are cleanly separated, swapping a model touches exactly one layer: the interface adapter. When they're fused, which is the default for code that grew organically, a swap touches all four, and that's the difference between an afternoon and a quarter.

The principle underneath all of this is the anti-corruption layer from domain-driven design: you build a boundary that translates the outside world's representation into your own, so changes outside don't ripple inward. Most agent codebases skip this boundary because the first provider's SDK is right there and works fine. The cost shows up only when you need a second provider.

The Model Abstraction Layer: What It Must Normalize

The heart of a portable agent is a thin abstraction over model calls. "Thin" is doing a lot of work in that sentence, the temptation is to build a fat abstraction that papers over every difference, and fat abstractions leak worse than thin ones. Aim for normalizing the things that genuinely differ in shape, not in semantics.

At minimum, your abstraction has to normalize:

• Message format. Most providers converged on a roughly OpenAI-shaped messages array, but the edges differ. Anthropic handles system prompts as a top-level parameter rather than a message role; some open-weight models served through various runtimes want raw chat templates. Your internal representation should be your own, a list of typed messages, and adapters translate to and from each provider. The Anthropic Messages API reference is a useful contrast point here precisely because it does not match OpenAI's shape.
• Token accounting and limits. Context windows, max output tokens, and tokenizer behavior vary. Your abstraction needs to expose these as model metadata so routing and truncation logic can reason about them.
• Stop reasons. "Stopped because it finished," "stopped because it wants to call a tool," and "stopped because it hit the token cap" are distinct states that every provider encodes differently. Normalize them into your own enum. Half the subtle bugs in multi-model agents come from misreading a stop reason.
• Streaming events. If you stream, the event shapes differ meaningfully. Normalize to your own event stream.
• Errors and rate limits. Retryable vs. terminal, rate-limit headers, overload signals, these belong in the adapter so your reliability infrastructure can apply uniform retries, fallbacks, and circuit breakers regardless of provider.

What you should not try to normalize is model capability. If one model supports parallel tool calls and another doesn't, hiding that behind a uniform interface produces an agent that behaves unpredictably depending on what's plugged in. Expose capabilities as metadata and let the orchestration layer adapt.

Tool-Calling Is Where Portability Breaks

If there's one place where multi-model agents quietly fail, it's tool-calling. Every provider has a function-calling format, they're all similar, and similar is more dangerous than different because it lulls you into thinking a naive adapter is enough.

The differences that actually bite:

• Schema dialects. Most providers accept JSON Schema for tool parameters, but they enforce different subsets. One rejects oneOf, another ignores format, a third silently drops descriptions past a certain length. Keep tool schemas in your own canonical form and generate provider-specific schemas at adapter time, validating against each provider's known constraints.
• Parallel vs. sequential calls. Some models emit multiple tool calls in one turn; others emit one at a time. Your loop has to handle both, which means designing the loop for the parallel case and degrading gracefully to sequential.
• Tool-result formatting. How you feed a tool's output back differs, a dedicated tool-result message type in one API, a specially formatted user message in another. Get this wrong and the model can't see its own tool results, which manifests as the agent repeating calls or hallucinating outcomes.
• Reliability of the call itself. Models differ in how often they produce malformed arguments or hallucinate tools that don't exist. This is the core concern of tool-calling reliability at the infrastructure layer, and it's model-specific: your validation and repair logic has to be strict enough to catch the worst-behaving model you might route to, not just your primary.

The pragmatic move is to treat tool-calling as a first-class part of your abstraction, not an afterthought. Define tools once in a canonical schema, run them through a translation and validation step per provider, and put a strict parser-and-repair layer between the model's raw output and your executor. The emerging Model Context Protocol standardizes tool definitions across hosts, which helps, though as the MCP standard explainer for operators notes, adoption is uneven and you still need your own validation boundary.

Routing: Choosing the Right Model Per Task

Once the model is genuinely swappable, you've earned the ability to use more than one at a time. This is where portability stops being defensive and starts paying for itself.

The basic pattern: route cheap, capable-enough models to easy steps and reserve frontier models for the hard ones. A classification step, a routine extraction, a "summarize this tool output" call, these don't need your most expensive model. The reasoning-heavy planning step might. This is the whole premise of the model-routing layer, and a portable agent is the prerequisite for doing it at all.

Routing strategies, roughly in order of sophistication:

• Static per-step routing. You hardcode which model handles which step. Simple, predictable, and 80% of the value. Start here.
• Capability-based routing. Route based on declared model metadata, long-context tasks to long-context models, tool-heavy tasks to models with strong function-calling.
• Cost-and-latency routing. Pick the cheapest model that clears a quality bar for the task class, with a fallback chain when the cheap model fails validation.
• Quality-tiered fallback. Try the cheap model; if its output fails your validators, escalate to a stronger one. This caps cost while protecting quality, and it's the pattern most worth building once your evals are solid.

A word of caution that doesn't get said enough: routing introduces non-determinism that's hard to debug. When an agent's behavior changes, "which model handled that step, and why" needs to be answerable from your traces. Without that, multi-model routing turns your observability stack into a crime scene with no witnesses.

The Evaluation Harness You Have to Own

Here's the part teams skip and then regret: you cannot safely swap or route between models without an evaluation harness that tells you whether a swap is actually safe.

The reason is simple. Two models can have nearly identical benchmark scores and behave completely differently inside your specific agent loop, on your specific tasks, with your specific tools. Public benchmarks tell you almost nothing about whether Model B will hold up where Model A currently works. The only thing that does is running both through your own task suite and comparing outcomes that matter to you: task success rate, tool-call validity, cost per completed task, latency, and failure modes.

This harness is the single highest-leverage investment in a portable agent program, and it's the thing that converts "we're afraid to switch" into "we switched on Tuesday." Build it as a set of representative tasks with programmatic graders where you can and human or model-judge graders where you can't. Run any candidate model through it before it touches production traffic. Re-run it whenever a provider ships a new snapshot, because "the same model name" is not the same model, providers update weights behind stable-looking identifiers, and your agent can regress silently.

Google's research teams and others have repeatedly shown that small prompt and model changes produce outsized, non-intuitive behavior shifts in agentic settings; the practical takeaway, echoed in industry guidance like a16z's writing on the emerging LLM app stack, is that your own evals are the only ground truth that counts. Treat the harness as core infrastructure, version it alongside your agent, and wire it into your agent CI/CD pipeline so a model swap is a tested change, not a prayer.

Build vs. Buy: Gateways, Frameworks, and Rolling Your Own

You don't have to build all of this from scratch. There's a spectrum of off-the-shelf help, each with a portability tradeoff.

Model gateways (the LiteLLM-style proxy, plus various commercial routers) give you a single OpenAI-compatible endpoint that fans out to many providers, handling translation, retries, and sometimes routing and caching. These are a fast way to get a normalized model interface, and they're genuinely useful. The catch: they normalize to a lowest-common-denominator shape, usually OpenAI's, which can mask provider-specific capabilities you actually want and quietly recreate lock-in to the gateway itself. Use them, but keep your own canonical message and tool representation behind them so the gateway stays swappable too.

Agent frameworks (LangChain, LlamaIndex, and the newer challengers) ship model abstractions as part of a larger orchestration package. They get you moving fast, but you inherit their abstraction's leaks and their choices about how the loop works. The framework-vs-platform strategic choice and the broader framework wars are worth thinking through before you commit, because the framework's model abstraction is now your model abstraction.

Rolling your own thin adapter layer is more work upfront but gives you the cleanest boundary and the least surprise. For a serious GaaS product where model portability is a competitive feature, where you're selling reliability and cost-efficiency to customers, owning this layer usually pays off. For an internal tool, a gateway plus a light wrapper is often the right call.

The honest heuristic: buy the gateway, own the canonical representation and the eval harness, and be skeptical of any framework that wants to own your control loop.

The Economics of Portability

Portability has a real engineering tax. Building and maintaining adapters, a canonical schema layer, routing logic, and an eval harness is weeks of work you could spend shipping features. It's fair to ask whether it's worth it.

The case for paying the tax:

• Pricing leverage. When you can credibly switch, you can negotiate, and you can chase the price-performance frontier as it moves, which it does, fast. Frontier model prices have fallen by roughly an order of magnitude per unit of capability on a regular cadence, and the agent economics only work if you can capture those drops.
• Risk insurance. Providers deprecate snapshots, change rate limits, suffer outages, and shift terms. A portable agent treats any single provider outage as a fallback event, not an incident. This is the same logic behind reliability infrastructure, redundancy at the model layer is just business continuity.
• Right-sizing. Multi-model routing typically cuts inference cost meaningfully by sending the bulk of cheap steps to cheap models. The savings often dwarf the cost of building the routing layer within a quarter at any real volume.

The case against, which is real: if you're pre-product-market-fit, building portability infrastructure before you know your agent even works is premature. Ship on one model, get it working, prove the loop. But design the boundaries cleanly from day one so that adding portability later is an extension, not a rewrite. The expensive mistake isn't skipping portability early, it's fusing the model into the loop so tightly that you can never cleanly extract it.

Insights Most People Overlook

The lock-in is in the prompts, not the API. Everyone obsesses over the API client as the switching cost, but the real cost is the prompt engineering tuned to one model's idiosyncrasies. A prompt that makes Model A reliable can make Model B flaky, because each model interprets instructions differently. Externalize prompts into a versioned registry with per-model variants from the start. A "model swap" is really a "prompt and model swap," and teams that don't plan for the prompt half get blindsided.

Gateways can recreate the lock-in they promise to solve. Routing everything through a single proxy that normalizes to OpenAI's shape feels like freedom, but you've just made the gateway your new single point of dependency, and lost access to provider-specific features it doesn't expose. Portability means the gateway is swappable too, which only holds if your own canonical representation sits behind it.

"Same model name" is not the same model. Providers update weights behind stable identifiers, and agents regress silently when that happens. Your eval harness isn't just for switching providers, it's a regression detector for your current provider's invisible updates. Teams that only run evals during migrations get surprised by quality drops they can't explain.

Capability normalization is an anti-pattern. The instinct to make every model look identical behind one interface actually hurts you. Parallel tool calls, long context, structured output guarantees, these differ, and hiding the differences produces an agent that behaves unpredictably depending on what's plugged in. Expose capabilities as metadata; adapt the loop to them.

Portability is a sellable feature, not just a hedge. In the GaaS market, "we route across models for cost and reliability" is a value proposition customers will pay for, not just an internal hygiene practice. The same architecture that protects you from lock-in lets you offer per-outcome pricing with healthier margins, because you control where each token lands.

References

• Anthropic, Messages API Reference
• Model Context Protocol, Tools Concept Documentation
• Andreessen Horowitz, Emerging Architectures for LLM Applications