Centralize or Federate? How to Decide Where Your AI Agents Actually Live

Table of Contents

• The Question Behind the Question
• What "Centralize" and "Federate" Actually Mean for Agents
• The Case for Centralizing
• The Case for Federating
• The Decision Framework: Five Forces That Decide for You
• Force 1: Regulatory Exposure
• Force 2: Workflow Heterogeneity
• Force 3: Program Maturity
• Force 4: Blast Radius
• Force 5: Talent Distribution
• The Hub-and-Spoke Model in Practice
• What Centralizes Well vs. What Federates Well
• How the Answer Changes as You Scale
• Insights Most People Overlook
• References

The Question Behind the Question

Ask a CIO whether agents should be centralized or federated and you'll usually get a philosophical answer rooted in how they ran their last big technology program, the cloud migration, the data platform, the RPA rollout. That instinct is worth something, but agents break the analogy in one important way: they act. A misconfigured dashboard shows wrong numbers. A misconfigured agent files a refund, sends an email to a customer, or updates a record in a system of record. The cost of a bad decision moves from "embarrassing" to "operational," and that single fact reshapes the centralize-versus-federate debate.

So the real question isn't "where should agents live?" It's "where should each decision about an agent live?" Who gets to choose which models an agent can call? Who approves the tools and APIs it can touch? Who watches it in production? Who buys the next one? Lumping all of those under one "centralize or federate" verdict is where most org designs go wrong. The good ones disaggregate the decision and route each piece to the level that can actually make it well.

What "Centralize" and "Federate" Actually Mean for Agents

In a centralized model, a single team, often an emerging AgentOps or AI platform group, builds, deploys, and operates agents on behalf of the whole company. Business units submit requests; the central team delivers. Think of it like an internal agency.

In a federated model, individual departments build and run their own agents on shared infrastructure and shared rules. Marketing runs its agents, finance runs its agents, and a thin central layer sets the standards everyone follows. Think of it like an internal marketplace with a building code.

Neither is the same as decentralized chaos, which is what you get when there are no shared rules at all and every team buys whatever agent vendor a sales rep emailed them last Tuesday. That's not a model; that's the absence of one, and it's the natural state most companies drift into if nobody decides on purpose. Containing that drift is its own discipline, see the related work on agent sprawl in this cluster.

The centralize/federate spectrum is really about where authority sits, not where code runs. You can run every agent on one platform and still federate ownership. You can run agents on five different stacks and still centralize governance. Keep those two axes separate in your head and the design choices get a lot clearer.

The Case for Centralizing

Centralization wins on the things that get worse when they're duplicated. A central team builds deep expertise fast because it sees every use case. It negotiates better with vendors because it buys at volume, relevant when you're juggling per-task and per-outcome pricing across a portfolio of agents. It standardizes the boring-but-critical layer: identity, secrets management, logging, evaluation harnesses, rollback procedures. When a new model drops or a prompt-injection technique makes the rounds, a central team patches once instead of chasing thirty teams.

There's also a quality argument. Reliable agents are genuinely hard to build, handling tool errors, retries, hallucinated parameters, and human-in-the-loop fallbacks is specialist work. Spreading that work across every department means every department reinvents the same failure modes, usually badly. McKinsey's research on scaling generative AI has consistently found that the organizations capturing real value treat it as a centrally-orchestrated capability rather than a thousand local experiments, precisely because the reusable plumbing dwarfs the per-use-case logic (McKinsey, "The state of AI").

The cost of centralizing is speed and fit. A central team becomes a queue. The marketing team that knows exactly what it wants waits behind finance's quarterly-close project, and the agent it eventually receives was built by someone who's never run a campaign. Centralization optimizes for control and economies of scale at the direct expense of responsiveness and domain nuance.

The Case for Federating

Federation wins on adoption and relevance, which, let's be honest, are the two things most agent programs are actually short on. The people closest to a workflow understand its edge cases, its unwritten rules, and its acceptable failure tolerances better than any central team ever will. When they own the agent, they iterate fast, they trust it more, and they're far more likely to actually delegate real work to it instead of treating it as a demo.

Federation also scales organizationally in a way centralization can't. A central team is a bottleneck by construction; there are only so many engineers, and demand for agents is going one direction. Push ownership to the edge and your capacity to build grows with the number of teams, not the size of one team. This is a big part of why smaller and mid-market companies often out-adopt enterprises, fewer layers between the person with the problem and the person who can ship the fix.

The cost is everything centralization was good at. Federated programs duplicate effort, drift on standards, and, most dangerously, create security and governance blind spots. When every team wires its own agent into production systems, you get inconsistent identity handling, untracked data access, and the lovely phenomenon of shadow agents: production-grade automations that the security team doesn't know exist until one of them does something it shouldn't. The governance literature is blunt that ungoverned autonomous systems are a material risk surface, and federation without strong central guardrails is how that risk compounds quietly (NIST AI Risk Management Framework).

The Decision Framework: Five Forces That Decide for You

Stop arguing about it in the abstract. Five concrete forces, scored honestly for your situation, will point you toward the right blend more reliably than any opinion.

Force 1: Regulatory Exposure

The more regulated the decision an agent influences, the more centralized its governance needs to be, full stop. An agent that touches patient data, makes credit determinations, or files anything with a regulator cannot have its guardrails set by whoever's running that department this quarter. Centralize the controls hard here even if you federate the build. In low-stakes domains (internal knowledge lookup, drafting, scheduling), the regulatory force is weak and you can let the edge run free.

Force 2: Workflow Heterogeneity

If every department's work looks broadly similar, centralization pays off, one well-built agent pattern covers many teams. If the work is wildly different from department to department, a central team will spend its life learning context it'll never fully grasp, and federation wins. Manufacturing-plus-retail-plus-R&D under one roof? High heterogeneity, lean federated. A company of near-identical regional branches? Low heterogeneity, lean centralized.

Force 3: Program Maturity

Early on, centralize. When you have three agents and no playbook, you need a single team to figure out what "good" even looks like, what to log, how to evaluate, when to keep a human in the loop. Federating before you have standards just multiplies your mistakes. As the program matures and the patterns harden into reusable platform and policy, you can safely push ownership outward. Maturity is the force most often ignored, and getting its timing wrong is why programs stall in pilot purgatory.

Force 4: Blast Radius

Score each agent by what it can break. An agent that only reads and drafts has a small blast radius, federate it cheerfully. An agent with write access to your ERP, the power to move money, or the ability to message customers at scale has a large one, its deployment, permissions, and monitoring belong under central control regardless of who built it. Blast radius should be assessed per-agent, not per-department, which is why a single team often ends up running a mix of centralized and federated agents.

Force 5: Talent Distribution

Be honest about where your agent-building skill actually sits. If it's concentrated in one team, you're effectively centralized whether you like it or not, and pretending otherwise just sets edge teams up to fail. If capable builders are spread across the business, increasingly common as internal-tools people pivot into agent work, federation becomes viable and central insistence on owning everything becomes pure bottleneck. Design for the talent you have, then invest deliberately to change the distribution you want.

The Hub-and-Spoke Model in Practice

Run those five forces and almost everyone lands in the same neighborhood: a hub-and-spoke model, sometimes called a center of excellence with federated execution. It's popular because it's correct for the median enterprise, not because it's a buzzword.

The hub owns what must be consistent: identity and access for agents, the shared observability and evaluation stack, the security guardrails and prompt-injection defenses, the procurement and vendor relationships, the model gateway, and the policy that says what agents may and may not do. The hub also runs the highest-blast-radius agents directly. Crucially, the hub's job is to make the right thing the easy thing, to ship paved-road tooling so good that edge teams want to use it rather than routing around it. A center of excellence that only says "no" gets bypassed within a quarter.

The spokes, the business units, own what must be local: the agent logic for their workflows, the iteration loop, the day-to-day operation and human oversight, and the relationship with the people whose work the agent touches. They build on the hub's platform and inside the hub's policy, but they own the outcomes.

The boundary between hub and spoke is where this model lives or dies. Draw it too far toward the hub and you've recreated the central bottleneck with extra steps. Draw it too far toward the spokes and the hub becomes a powerless committee while sprawl returns. Gartner's analysts have repeatedly framed this kind of federated-with-strong-center arrangement as the durable operating model for scaled AI, precisely because it lets you tune the boundary as conditions change (Gartner on AI operating models).

What Centralizes Well vs. What Federates Well

A practical cut, because the abstract framework is only useful if you can apply it Monday morning:

Centralize these: agent identity and credentials; the model gateway and approved-model list; security guardrails and red-teaming; the observability, logging, and evaluation platform; vendor procurement and contract terms; high-blast-radius agents; the governance policy itself; and incident response when an agent misbehaves.

Federate these: agent logic and prompts for domain-specific workflows; the build-and-iterate loop; routine human-in-the-loop oversight; low-blast-radius agents; feedback collection from end users; and the prioritization of which workflows to automate next.

Negotiate these (they shift with maturity): who can approve a new tool integration; who signs off on production deployment; how much budget sits centrally vs. with units; and how much standardization to enforce on agent frameworks. Early-stage programs pull these toward the center; mature programs push them out.

Notice that the centralize list is dominated by security, identity, and economics, while the federate list is dominated by domain logic and iteration. That's not a coincidence, it's the whole principle in one sentence. Centralize what's expensive to duplicate and dangerous to get wrong; federate what's expensive to coordinate and cheap to get wrong locally.

How the Answer Changes as You Scale

The single biggest mistake in this whole topic is treating the centralize/federate decision as permanent. It isn't. The right model migrates along a predictable arc, and the teams that win are the ones who move deliberately instead of getting stuck.

One to a few agents: centralize almost everything. You're learning what good looks like, and a single team needs to set the patterns. Federating now just scatters your few hard-won lessons.

Scaling to a fleet: shift to hub-and-spoke. The central team stops being the builder-of-everything and becomes the platform-and-policy team. This transition is uncomfortable, the central team has to give up control it spent the early phase accumulating, and a surprising number of programs stall right here because nobody wants to let go.

Many agents across many units: the hub gets thinner and more platform-like; the spokes get stronger. The hub's value shifts from "we build your agents" to "we make sure your agents are safe, observable, and economical to run." Vendor management, cost control across dozens of agents, and sprawl containment become the hub's main job, a long way from where it started.

The forces that pointed you toward a model in year one will have shifted by year three: maturity rises, talent spreads, the regulatory picture sharpens. Re-score them annually. The organizations that treat their operating model as a living decision rather than a one-time architecture choice are the ones that don't have to blow it all up and start over.

Insights Most People Overlook

The centralize/federate debate is usually a proxy for an unspoken budget fight. When IT wants to centralize and the business wants to federate, the technical arguments are often cover for "who controls the spend and the headcount." Name that out loud and the conversation gets dramatically more productive. A lot of "architecture disagreements" are really org-chart disagreements wearing a lab coat.

You can, and usually should, centralize and federate the same agent on different axes. A finance agent can have centrally-owned identity and guardrails, federated logic and ownership, and a deployment approval that's negotiated between them. People treat it as a binary per agent; the sophisticated move is to slice each agent across the five forces independently. Almost no one designs at that resolution, which is exactly why it's an edge.

Federation fails silently; centralization fails loudly. A centralized program's failure is visible, the queue backs up, teams complain, the bottleneck is obvious and gets escalated. A federated program's failure is invisible until something breaks: the shadow agents, the inconsistent permissions, the security gap nobody flagged. This asymmetry matters because it means centralized failures get fixed and federated failures accumulate. If you federate, you have to invest in making its failure modes loud, central observability over every agent, sprawl audits, mandatory registration, or you won't see the problem until it's a headline.

The "center of excellence" is a phase, not a permanent fixture, and it should plan its own obsolescence. The healthiest central teams are the ones actively working to push more ownership outward over time, measuring their success by how little they have to do directly. A CoE that's still hand-building every agent in year three has failed even if every agent works, because it's become the ceiling on how fast the company can adopt.

Small blast radius plus high heterogeneity is the strongest case for aggressive federation, and it's more common than people think. Most of the actual day-to-day agent opportunity in a company is low-stakes, department-specific drudgery: drafting, summarizing, looking things up, routing. That work scores small-blast-radius and high-heterogeneity, which is the federate-hard quadrant. Companies over-centralize because the scary agents grab attention, but the scary agents are the minority. Designing your whole model around them throttles the boring majority where most of the value actually is.

References

• McKinsey & Company, The state of AI
• NIST, AI Risk Management Framework
• Gartner, Artificial Intelligence insights and operating models