The Observability Data an Enterprise Buyer Demands Before Signing

Table of Contents

• Why Observability Became a Procurement Gate
• The Seven Data Artifacts Buyers Actually Ask For
• 1. Full Per-Run Traces
• 2. Task Success Rate With a Visible Denominator
• 3. Tool-Call and External-Action Logs
• 4. The Audit Trail and Who-Did-What Lineage
• 5. Drift and Regression Signals Over Time
• 6. Escalation and Human-Override Telemetry
• 7. Cost and Latency Per Outcome
• How to Read a Vendor's Observability Story
• What Buyers Should Put in the Contract
• Insights Most People Overlook
• References

Why Observability Became a Procurement Gate

A few years ago, buying software meant asking about uptime, SOC 2, and a roadmap. Buying an autonomous agent is different in a way that catches a lot of procurement teams off guard the first time: the thing you are buying makes decisions on its own, and those decisions touch your systems, your customers, and sometimes your money.

That changes the conversation. When a vendor sells you a per-outcome service, a few dollars per resolved support ticket, per reconciled invoice, per qualified lead, you are not buying a tool you operate. You are buying an outcome that an opaque system produces. And the first question any serious risk team asks is: how do I know what it did, and how do I prove it later?

That question is the whole ballgame. It is why observability has quietly moved from a "nice engineering hygiene" item to a hard gate in enterprise deals. Security teams have learned, often the hard way, that an agent that fails loudly is fine; you catch it. The dangerous one is the agent that fails quietly, the silent failure problem where it confidently does the wrong thing and reports success. You cannot catch that without data. And you cannot get that data after the fact if the vendor never captured it.

There is also a regulatory undertow here. Frameworks like the NIST AI Risk Management Framework explicitly call for traceability and documentation of AI system behavior, and the EU AI Act pushes record-keeping obligations onto high-risk systems. A buyer in financial services or healthcare is not asking for traces because they are curious. They are asking because their own auditors will ask them.

So the buyer shows up with a list. Here is what is on it.

The Seven Data Artifacts Buyers Actually Ask For

The strongest enterprise buyers I have watched do this diligence do not ask "is it observable?", a useless question that every vendor answers yes to. They ask for specific artifacts and then check whether the vendor can actually produce them on the spot. The difference between a vendor with real observability and one with a marketing dashboard shows up in about ninety seconds of follow-up questions.

1. Full Per-Run Traces

The bedrock request. For any single agent run, the buyer wants the complete trace: the input, every reasoning or planning step, every tool call, every intermediate result, and the final output. Not a summary. The raw, timestamped sequence of what the agent did, in order.

This is harder than it sounds, which is exactly why it is a good filter. Traditional application monitoring assumes deterministic code paths; an agent's path branches on model output, so a real trace has to capture the non-deterministic decision points, not just the function calls. Vendors who bolted a logging library onto a multi-step workflow tend to produce traces with holes, the tool calls are there but the reasoning that triggered them is missing, so you can see what happened but never why.

Ask for a trace of a failed run, specifically. Anyone can show you a clean happy path. The vendor who can pull up a failure from last Tuesday and walk you through where the agent went sideways is the one who actually instruments their system. The OpenTelemetry project has been extending its conventions to cover generative AI spans, and vendors leaning on that standard tend to have cleaner, more portable trace data than those rolling their own.

2. Task Success Rate With a Visible Denominator

Every vendor will quote you a success rate. "Our agent resolves 94% of tickets." The number is meaningless until you ask the only question that matters: 94% of what?

A success rate is a fraction, and the denominator is where the truth hides. Did they count only the tickets the agent attempted, quietly excluding the ones it punted to a human? Did "resolved" mean the customer was satisfied, or that the ticket was closed? Over what time window, and on whose data, a curated benchmark or your messy production reality?

Buyers who have been burned demand the definition in writing: what counts as a task, what counts as success, what counts as an attempt, and what gets excluded. Then they ask for the rate broken down by task type, because a blended 94% often hides a category where the agent is at 60% and the vendor would rather you not notice. This is the same reason "99% accurate" is essentially a meaningless claim for agents, accuracy without a denominator and a task taxonomy is a vanity metric.

3. Tool-Call and External-Action Logs

An agent that only talks is low-risk. An agent that acts, sends emails, updates records, moves money, files tickets, calls APIs, is where the real exposure lives. Buyers want a discrete, queryable log of every external action the agent took, separate from its reasoning trace.

Why separate? Because the security team's question is narrow and specific: show me every write operation this agent performed against our systems last month. They do not want to reconstruct that from prose traces. They want a table: timestamp, action, target system, parameters, result, and the run ID that triggered it. When a tool call fails in the middle of a chain, this log is also how you figure out whether the agent retried correctly or silently abandoned a half-finished operation, one of the nastier failure modes in production.

4. The Audit Trail and Who-Did-What Lineage

This is the artifact that wins deals in regulated industries, and the one most early-stage vendors fumble. An audit trail is not the same as logs. Logs are for engineers debugging. An audit trail is a tamper-evident, retained, exportable record designed to answer a question months later, possibly to a regulator or in a dispute: who or what took this action, on whose authority, with what inputs, at what time.

The lineage piece matters more as agents start calling other agents. In a multi-agent chain, when one weak agent breaks the result, the buyer needs to trace the bad output back through the chain to the originating decision. A flat log cannot do that; you need the parent-child relationship between runs preserved. Mature vendors treat the audit trail as a first-class product surface, with defined retention, immutability guarantees, and an export format that survives the vendor going out of business.

5. Drift and Regression Signals Over Time

Agents do not stay still. The underlying model gets updated by the provider, prompts get tuned, the world the agent operates in shifts, and quality erodes in ways that no single run reveals. Buyers who understand this ask a pointed question: how would I know if your agent got worse over the next six months?

The answer they want involves continuous evaluation, not a one-time pre-launch benchmark. Show me the success rate trended weekly. Show me how you detect drift when the agent slowly degrades. Show me what happens to your numbers when the model provider ships a new version underneath you, because regression testing agents across model changes is exactly the scenario that breaks naive vendors. McKinsey's research on scaling AI has repeatedly flagged that the gap between pilot and production is mostly an operational and monitoring problem, not a model-capability problem, and drift is the operational issue that bites hardest after the deal closes.

6. Escalation and Human-Override Telemetry

A well-designed agent knows its limits and escalates. The telemetry around that boundary is enormously revealing, and smart buyers mine it. They want: how often does the agent escalate to a human, on what triggers, and what happens after?

Two numbers tell you a lot. A high escalation rate might mean the agent is appropriately cautious, or that it is barely autonomous and you are paying outcome pricing for a glorified router. A low escalation rate paired with a high human-override rate is a red flag: it means the agent is plowing ahead on tasks it should have escalated, and humans are constantly cleaning up after it. The override log, every time a person reversed or corrected what the agent did, is one of the highest-signal datasets in the whole bundle, and it is the one vendors are most reluctant to share.

7. Cost and Latency Per Outcome

Per-outcome pricing only makes sense if both sides can see the unit economics. Buyers increasingly ask for cost and latency attributed to individual outcomes, not aggregate cloud spend. How long does a resolution actually take? What is the token and compute cost behind that flat per-task fee, and how does it move when tasks get harder?

This matters beyond budgeting. There is a real latency-reliability tradeoff in production agents, the cheap, fast configuration is often the less reliable one. A buyer who can see cost and latency per outcome can tell whether a vendor is quietly degrading quality to protect their margin on a per-outcome contract, which is a structural conflict of interest baked into the pricing model itself.

How to Read a Vendor's Observability Story

Once you know the seven artifacts, evaluating a vendor gets faster. A few patterns separate the real from the theatrical.

The dashboard tell. If the vendor's entire observability pitch is a slick dashboard, be skeptical. Dashboards aggregate. Aggregation hides the individual failures you most need to see. Ask to drill from a top-line number all the way down to a single raw trace. If you cannot, the dashboard is a veneer over data they may not actually retain.

The export tell. Ask for the raw data export format and the retention period. A vendor confident in their observability hands you an API and a schema. A vendor who can only show you data inside their own UI is a vendor whose data you do not really own, a problem the moment you want to run your own evals or the relationship ends.

The failure tell. Ask them to walk through a real incident: a time the agent did something wrong in production, how they detected it, and what the trace showed. Vendors with a genuine post-mortem culture for agent failures answer this comfortably. Vendors who claim their agent has never failed are either lying or not looking, and both are disqualifying.

The verification tell. Increasingly, mature vendors run a second verification layer that checks the agent's work with another agent or a deterministic rule set. Ask whether that verification output is itself observable. The check is only as trustworthy as your ability to see when the check disagreed with the primary agent.

What Buyers Should Put in the Contract

Diligence is worthless if the data evaporates after signing. The strongest enterprise contracts I have seen pin down observability as a contractual right, not a goodwill gesture. A few clauses worth fighting for:

A data access guarantee, the buyer gets API-level access to traces, action logs, and the audit trail for their own tenant, not just a read-only dashboard. A retention commitment, audit data kept for a defined period (often matching the buyer's own regulatory retention) and exportable on demand and at termination. A definitions appendix, the exact, frozen definition of "task," "success," and "attempt" used to compute any reliability SLA the vendor is promising, so the number cannot be quietly redefined later. And a regression-notification clause, the vendor must disclose material changes to the underlying model or prompts that could affect behavior, because a model swap underneath a stable-looking agent is invisible to the buyer otherwise.

None of this is exotic. It is the same instinct that made buyers demand uptime SLAs and breach-notification clauses a decade ago, applied to a system whose defining feature is that it acts on its own. The vendors who will win the GaaS market are not fighting these requests, they are showing up to the first meeting with the data already on the table, because they figured out early that in agentic AI, observability is the product as much as the agent is.

Insights Most People Overlook

The override log is more honest than the success rate. Vendors curate their success metrics; almost nobody curates the human-override log, because it is generated by their own customers correcting the agent in real time. If you can get access to it, the override rate by task type tells you the unvarnished truth about where the agent is actually weak, often a very different picture from the marketing deck.

Demanding observability you cannot consume is theater. Plenty of buyers extract a beautiful trace-export API in the contract and then never staff anyone to look at it. The data is only valuable if you have an internal eval or reliability function that ingests it. Buying the right to data you will never read just transfers the risk back onto yourself while letting you feel covered. Budget for the consumer of the data, not just the clause.

Per-outcome pricing creates a hidden incentive to under-instrument. A vendor paid per successful outcome has a quiet motive to define "success" loosely and to not log too carefully, because rich observability mostly exposes failures that cost them money. The pricing model and the observability quality are in tension. The vendors who instrument heavily despite per-outcome pricing are signaling real confidence, it is a costly, credible signal precisely because it works against their short-term margin.

The audit trail's real test is the vendor's death. Most observability discussions assume the vendor sticks around. The artifact that matters in a crisis is the one you can read without them. Ask explicitly: if your company disappears tomorrow, what format is my audit trail in, and can I read it with no software of yours? If the answer is a proprietary store you can only query through their portal, you do not have an audit trail, you have a dependency.

"No failures" is the worst possible answer. Buyers sometimes reward the vendor who claims a spotless record. It should be the opposite. A vendor with rich observability knows their agent fails sometimes, because they can see it. The claim of perfection is itself evidence they are not measuring. Reward the vendor who shows you their failures and their fixes over the one who insists there are none.

References

• NIST AI Risk Management Framework
• OpenTelemetry Semantic Conventions for Generative AI
• McKinsey QuantumBlack, Insights on Scaling AI