Gross Margin Math for an Agent That Calls Three Other Vendors' Models

Table of Contents

• The setup nobody puts in the pitch deck
• Walking the cost of a single task
• Step 1: the planner call
• Step 2: retrieval and embeddings
• Step 3: the generation call
• Why your COGS is a distribution, not a number
• The markup you're actually paying three vendors
• Retries: the line item that eats the margin
• A worked margin model you can copy
• Levers that actually move the number
• Insights Most People Overlook
• References

The setup nobody puts in the pitch deck

Picture a contract-review agent. A customer uploads a 40-page master services agreement and asks, "flag anything unusual versus our standard terms." That single request, one billable task, does not map to one model call. It maps to a small supply chain.

The agent reasons about what to do (a frontier model from Vendor A). It chunks and embeds the document to retrieve the customer's playbook (an embeddings model from Vendor B). It drafts the redline and explanation (a cheaper, fast model from Vendor C, because generation volume is high and you don't need the smartest model for every paragraph). Three vendors, three invoices, one price you charge the customer.

Founders love to say "our gross margin is 70%." But 70% on what? On the median task? The 90th-percentile task? The task where the planner decided to re-read the document twice? In a multi-vendor agent, your cost of goods sold isn't a constant you multiply against revenue. It's an emergent property of how the orchestration behaves on real inputs. That's the whole game, and it's why this article exists as a node in the broader GaaS economics cluster, the unit-cost mechanics here feed directly into how you price, forecast, and value the business.

Let me walk an actual task end to end, then build the margin model around it.

Walking the cost of a single task

I'll use representative 2026 list prices. Plug in your own; the structure is what matters. Assume the agent processes the 40-page contract, roughly 30,000 input tokens of document plus prompts.

Step 1: the planner call

The planner is a strong reasoning model. It reads a summary of the task, decides the workflow, and may emit a chain of thought before producing a plan. Say it consumes 8,000 input tokens and produces 2,000 output tokens, including reasoning tokens you pay for but never show the user.

At a representative frontier rate of $3 per million input and $15 per million output:

• Input: 8,000 × $3 / 1,000,000 = $0.024
• Output: 2,000 × $15 / 1,000,000 = $0.030
• Planner subtotal: $0.054

Cheap in isolation. Watch what happens when it isn't called once.

Step 2: retrieval and embeddings

To compare against the customer's standard terms, you embed the contract's clauses and run similarity search against a stored playbook. Embeddings are dirt cheap per token, on the order of $0.02-$0.13 per million tokens depending on the vendor and model, so embedding 30,000 tokens costs a fraction of a cent. Call it $0.002. Vector search itself is mostly your own infra cost, not a vendor pass-through, so it lands in a different bucket (one worth tracking separately, as the cluster piece on shared infrastructure cost attribution argues).

• Retrieval subtotal: ~$0.002 in vendor model cost

This step is almost free at the token level, which is exactly why people ignore it. It still adds latency and a third dependency to your reliability surface.

Step 3: the generation call

Now the agent writes the redline. This is the token-heavy step. It re-ingests relevant clauses, the retrieved playbook passages, and instructions, say 20,000 input tokens, and produces a detailed 6,000-token explanation. You deliberately route this to a cheaper, fast model at roughly $0.50 input / $1.50 output per million:

• Input: 20,000 × $0.50 / 1,000,000 = $0.010
• Output: 6,000 × $1.50 / 1,000,000 = $0.009
• Generation subtotal: $0.019

Add it up. A clean, single-pass task costs about $0.075 in vendor model fees. If you charge $2.00 per contract reviewed, that's a 96% gross margin on the happy path. Spectacular. File it under "numbers that won't survive contact with production."

Why your COGS is a distribution, not a number

The $0.075 above assumes every step fires exactly once and the document is average-sized. Neither holds. Real cost per task is shaped by four sources of variance, and they compound:

1. Input size variance. A 5-page NDA and a 200-page credit agreement are the same "task" to your pricing page but a 40× difference in token cost. If your customers skew long, your median quietly drifts up.
2. Fan-out. The planner may decide the contract needs clause-by-clause analysis, spawning a generation call per section. One task becomes thirty calls. This is the core mechanic behind fan-out economics when agents spawn sub-agents, and it can turn a $0.075 task into a $2.00 task that wipes out the margin entirely.
3. Retries. A malformed tool call, a timeout, a JSON parse failure, or a self-critique loop re-runs steps. More on this below, it's the single most underestimated line item.
4. Vendor price drift. Three vendors means three independent pricing roadmaps. One raising output prices, or you migrating to a newer (pricier) model for quality, shifts your COGS without any change in your product.

So the right artifact isn't "gross margin = X%." It's a histogram of cost per task across a real sample, with a p50, p90, and p99. Price against the p90, not the p50, or your worst customers, the ones with the longest documents and the messiest inputs, will be the ones you lose money serving. The mechanics of why averages mislead here are covered well in a16z's writing on the economics of AI agents and why per-call costs deceive.

The markup you're actually paying three vendors

Here's the part founders rationalize away. When you call three vendors' hosted APIs, you are paying each one's gross margin on top of raw GPU cost. Frontier labs run inference at healthy markups; independent analyses have pegged hosted-API gross margins in the broad 50-75% range depending on model and load, though the labs don't publish exact figures. You're stacking three of those markups into one task.

This is the case the "we'll just orchestrate other people's models" crowd has to answer. Your COGS is structurally someone else's revenue, three times over. That's fine, plenty of great businesses resell compute, but it caps your ceiling and makes you fragile to their pricing decisions. It's the same dynamic the cluster piece on the margin trap of passing through model costs dissects: pass-through pricing feels safe but transfers all the upside to your vendors.

Two structural responses, neither free:

• Substitution. Move the high-volume, low-judgment steps (generation, classification) to cheaper or open-weight models you host yourself, paying GPU cost instead of API markup. This is where the open-weight vs. frontier model economics question gets real. You trade vendor markup for ops burden and undifferentiated heavy lifting.
• Negotiation and committed-use discounts. At volume, the labs cut deals. But discounts on three separate contracts are harder to land and easier to lose than one big commitment, and committed spend introduces its own reserved-vs-on-demand tension.

Cloud taught this lesson a decade ago: when compute is your COGS, you obsess over rate, utilization, and commitment structure. Andreessen Horowitz's older but still-cited analysis of cloud repatriation and the cost of compute at scale reads almost word-for-word as a warning to GaaS founders.

Retries: the line item that eats the margin

Run the single-task math again, but assume realistic failure behavior. Suppose:

• The planner succeeds first try (95% of the time).
• The generation step produces invalid output 15% of the time and gets retried once.
• A self-critique pass runs on 30% of tasks, adding a full extra generation call.

Now the expected cost of a task isn't $0.075. The generation step's expected cost is no longer $0.019, it's $0.019 × (1 + 0.15 retry + 0.30 critique) = roughly $0.028. Small in absolute terms here, but on agents where the heavy model does the generating, a 45% inflation of the dominant cost line is the difference between 70% and 55% gross margin.

The deeper trap: retries correlate with the hard inputs you can least afford to lose money on. The 200-page agreement is both the most expensive task and the most likely to trigger retries and critique loops. Your cost variance and your input variance point the same direction. This is why the hidden cost of retries, when one task becomes fifty model calls deserves its own treatment; here it's enough to say: model retries explicitly, with a probability and a cap, or your margin model is fiction.

A worked margin model you can copy

Build a per-task cost as the sum of expected vendor costs across steps, then derive margin against your price. The skeleton:

cost_per_task =
 P(planner_called) × planner_cost × E[planner_runs]
 + P(retrieval_called) × retrieval_cost
 + P(generation_called) × generation_cost × E[generation_runs]
 + fixed_infra_cost_per_task # vector DB, orchestration, logging

gross_margin = (price - cost_per_task) / price

Then compute it three times, at p50, p90, and p99 input size and retry behavior, using a real sample of tasks, not assumptions. A representative output might look like:

Scenario	Vendor cost/task	Price	Gross margin
p50 (clean, average doc)	$0.09	$2.00	95.5%
p90 (long doc, one retry)	$0.41	$2.00	79.5%
p99 (huge doc, fan-out + critique)	$1.85	$2.00	7.5%

That p99 row is the whole point. A handful of tasks can run at near-zero or negative margin, and if those tasks cluster in one or two heavy customers, you have a cost-attribution problem, you're funding a money-losing account with everyone else's margin. The fix isn't always raising prices. Sometimes it's a fan-out cap, a per-task token budget, or routing the heaviest documents to a batch tier. Several startups are quietly doing exactly this; it's the subject of the cluster piece on capping autonomy to protect margin.

Levers that actually move the number

Once you can see the distribution, the optimization order is fairly consistent across multi-vendor agents:

• Route by difficulty, not by default. Most steps don't need your smartest (priciest) vendor. A cheap model for generation and a frontier model only for the planner is the single biggest lever, and it's why model routing is becoming a discipline of its own.
• Cache aggressively. Prompt caching on repeated system prompts and retrieved context can cut input token cost on the dominant step substantially. Caching and memory are quiet margin levers most teams underuse, worth a dedicated look.
• Cap fan-out and tokens per task. A hard ceiling converts an unbounded p99 tail into a known worst case. You lose a little quality on pathological inputs; you gain a margin floor you can actually price against.
• Kill needless retries and critique loops. Measure how often the second pass actually changes the answer. If self-critique improves output 4% of the time but runs 30% of the time, you're paying a lot for a little.
• Watch all three price sheets. Set an alert on each vendor's pricing and on your blended cost per task. Token prices are volatile week to week, and a single vendor's change can move your blended COGS without warning.

The uncomfortable truth: in a three-vendor agent, you don't fully control your COGS. You control routing, caching, retries, and caps. Those four levers are your real margin surface. Everything else is somebody else's pricing committee.

Insights Most People Overlook

• Your cheapest vendor sets your reliability ceiling, not your cost floor. Teams route high-volume generation to the cheapest model to save money, then eat the savings, and more, in retries when that model fails structured-output validation more often. Cheap-per-token and cheap-per-successful-task are different metrics, and the gap is exactly your retry rate. Optimize for cost per completed task, not cost per call.
• Embeddings are a rounding error in dollars but a first-class citizen in risk. The retrieval step costs fractions of a cent, so nobody models it. But it's a third vendor in your critical path: a third API that can rate-limit you, deprecate a model, or have an outage. Your margin model ignores it; your incident postmortem won't.
• Reasoning tokens are invisible COGS. When the planner "thinks" before answering, you pay for output tokens the customer never sees. On reasoning-heavy models this can be the majority of a task's planner cost. It doesn't appear in your output, so it doesn't appear in most people's cost spreadsheets, until the invoice does.
• Per-outcome pricing inverts the margin risk you just modeled. Everything above assumed per-task pricing, where the customer absorbs input variance. Flip to per-outcome (charge only when the contract is successfully reviewed and accepted) and the long, retry-prone, fan-out-heavy tasks, the expensive ones, become the tasks you might not get paid for at all. The p99 cost row doesn't go away; it just moves from "thin margin" to "unbilled."
• Three vendors is a hedge and a hostage situation at once. Multi-vendor orchestration protects you if one lab degrades or hikes prices, you can re-route. But it also means any one of three vendors can break your unit economics unilaterally, and you're too small to matter to all three at once. Single-vendor agents have worse diversification and better leverage. Pick deliberately.

References

• The Economic Case for Generative AI and Foundation Models, Andreessen Horowitz
• The Cost of Cloud, a Trillion Dollar Paradox, Andreessen Horowitz