Vector Databases in the Agent Stack: Still Necessary, or Already Legacy?

Table of Contents

• The 2023 Assumption Everyone Inherited
• What Actually Changed
• Context Windows Ate the Easy Cases
• Hybrid Search Quietly Won
• Agentic Retrieval Replaced One-Shot Lookup
• Where Vector Databases Still Win
• Where They Don't (And You're Paying Anyway)
• The Economics for a GaaS Operator
• A Decision Framework You Can Actually Use
• Insights Most People Overlook
• References

The 2023 Assumption Everyone Inherited

If you built anything with LLMs two years ago, the reference architecture was practically a meme: chunk your documents, embed the chunks, shove the vectors into Pinecone or Weaviate or pgvector, embed the user's query, run an approximate-nearest-neighbor search, stuff the top-k results into the prompt. RAG. Done. Every tutorial showed the same diagram, and every diagram had a cylinder labeled "vector DB" sitting dead center.

That assumption hardened into infrastructure. A whole funding cycle ran on it, vector database startups raised at valuations that priced in "every AI app needs one of these." And for a moment, that was nearly true, because the alternative was a 4,000-token context window that couldn't hold a single SEC filing, let alone a knowledge base.

The trouble is that the agent stack of 2026 looks almost nothing like the chatbot stack of 2023, and a lot of the load-bearing assumptions never got re-examined. Agents don't do one-shot retrieval into a frozen prompt. They loop. They call tools. They decide what to fetch next based on what they just learned. The vector database was designed for a request/response world, and agents don't live there anymore.

What Actually Changed

Three things shifted underneath the old architecture, and each one chipped away at the vector database's default position.

Context Windows Ate the Easy Cases

When the usable context window was a few thousand tokens, you had to retrieve. There was no choice, the knowledge didn't fit. Now frontier models routinely handle hundreds of thousands of tokens, and the practical ceiling keeps climbing. Anthropic's own documentation on long context and prompt caching shows you can park a large, stable corpus in the prompt, cache it, and pay a fraction of the cost on every subsequent call.

For a surprising number of use cases, a support agent that needs your full product manual, a contract assistant that works within a single 80-page agreement, the entire relevant corpus now fits in context. When that's true, a vector database doesn't improve accuracy; it degrades it, because chunking throws away document structure and top-k retrieval can silently drop the one paragraph that mattered. "Just put it all in the window and cache it" is now a legitimate, often superior architecture for the small-corpus case, and it has zero vector infrastructure.

This doesn't scale to a 50-million-document enterprise corpus, obviously. But it kills the easy 60% of cases that used to be the vector DB's bread and butter.

Hybrid Search Quietly Won

Here's the uncomfortable empirical finding that the embedding-everything crowd doesn't love: pure semantic search frequently loses to plain old keyword search, or to a hybrid of both. When a user searches for an exact error code, a product SKU, a person's name, or a specific API method, lexical matching (BM25) nails it and dense embeddings often whiff, because the embedding "understands" the query's meaning but blurs the exact token you needed.

The industry has largely converged on hybrid retrieval, combine sparse lexical scores with dense vector scores and rerank, as the actual best practice, a point reinforced across vendor and research write-ups on retrieval-augmented generation patterns. And once you're running hybrid search with a reranker, the vector component is one signal among several, not the spine of the system. Postgres with pgvector plus full-text search, or an Elasticsearch/OpenSearch cluster doing both, handles this without a dedicated vector database at all.

Agentic Retrieval Replaced One-Shot Lookup

This is the big one, and it's specific to the agent context that defines this cluster. Classic RAG is one embed, one search, one answer. An agent doesn't work that way. It issues a query, reads the result, realizes it needs something adjacent, issues another, follows a citation, calls a structured API to get the authoritative number rather than a fuzzy semantic match, and only then answers.

When retrieval becomes an iterative, tool-driven loop, closely related to retrieval for agents beyond basic RAG and the broader move toward agentic search, the value shifts away from "one great similarity score" toward "a flexible set of retrieval tools the agent can compose." A vector search can be one of those tools. But so can a SQL query, a graph traversal, a keyword index, and a plain API call. The agent's reasoning loop is now doing the work that a clever retrieval pipeline used to do, which means the retrieval layer can be dumber and more heterogeneous.

Where Vector Databases Still Win

I want to be fair, because there's a contrarian-to-the-contrarians point here: people declaring vector databases obsolete are overcorrecting. There are workloads where nothing else comes close.

The clearest case is large-scale semantic recall over a corpus too big for any context window and too unstructured for keyword search to help. Think millions of support tickets, a decade of internal wiki pages, a media archive, a sprawling codebase. When a user asks "have we seen anything like this before?" and "like this" is a genuinely fuzzy, conceptual match across millions of items, dense vector search is the only primitive that works at that scale and latency. BM25 can't find conceptual similarity; the context window can't hold the corpus.

The second case is cross-modal and recommendation-style retrieval, matching images to text, finding similar user behaviors, deduplicating near-identical content. These are native vector problems and aren't going anywhere.

The third, underrated case is agent long-term memory at scale. This connects directly to memory systems for agents: when an agent accumulates thousands of past interactions and needs to recall "the relevant prior episode," semantic similarity over an embedding store is a reasonable mechanism, provided you've thought hard about what should be embedded versus what belongs in structured storage. (Most teams over-embed memory; more on that below.)

Where They Don't (And You're Paying Anyway)

The failure mode I see most often: a team stands up a managed vector database on day one because the tutorial said to, then builds an agent whose actual corpus is 200 documents that comfortably fit in a cached context window. They're now paying for a hosted vector service, an embedding pipeline, a re-indexing job, and the operational burden of keeping embeddings in sync with a source of truth, to solve a problem they don't have.

Other common mismatches:

• Small or slowly-changing corpora. If it fits in context, cache it. The vector DB adds latency, cost, and a chunking-induced accuracy tax.
• Queries that are mostly exact-match. Internal tools where people search by ID, name, or code. BM25 wins; embeddings add noise.
• Anything needing precise, current, structured data. Inventory counts, account balances, prices. The agent should call an API or query a database, not retrieve a stale embedded snapshot. Semantic similarity to a number is meaningless.
• Single-document reasoning. Summarizing or Q&A over one contract, one report, one thread. Chunking actively hurts; whole-document context wins.

In all of these, the vector database isn't just unnecessary, it's a source of bugs (the silent top-k miss) and a recurring cost line for a GaaS operator trying to hit per-task margins.

The Economics for a GaaS Operator

This cluster is about agents sold as a service, where the unit economics of a single task determine whether the business works. So the vector-database question is ultimately a cost-and-reliability question, and it ties into the GaaS infrastructure cost stack, decomposed.

A dedicated managed vector database is a fixed-ish cost: you provision capacity, you pay for storage and the index sitting in memory, you pay for the embedding compute to ingest and re-embed. That cost exists whether or not a given task touches it. For per-outcome pricing, fixed infrastructure that doesn't scale down with idle periods is exactly the kind of overhead that quietly eats margin. Industry analyses of agent infrastructure economics, see a16z's writing on the emerging AI infrastructure stack, keep landing on the same theme: the winning architectures push as much as possible into variable, per-call costs and ruthlessly cut standing infrastructure.

Against that, prompt caching has changed the math. Caching a large stable corpus in context turns a "fixed vector index + embedding pipeline" cost into a "discounted per-call cache read" cost that scales with actual usage. For corpora in the range that fits, this is frequently cheaper and more accurate. The vector database earns its fixed cost only when the corpus is large enough that you genuinely cannot cache it, at which point it's not overhead, it's the only thing that works.

The practical upshot: don't default to a vector database, and don't ban it either. Provision it when the corpus crosses the threshold where context caching and hybrid keyword search stop being viable. Below that threshold, you're burning margin on infrastructure your agent doesn't need.

A Decision Framework You Can Actually Use

When someone hands you an agent spec, walk these questions in order:

1. Does the entire relevant corpus fit in a cached context window? If yes, start there. No vector DB. Add retrieval only if accuracy or cost data tells you to.
2. Are the queries mostly exact-match (IDs, names, codes, errors)? If yes, you need lexical/keyword search, not embeddings. pgvector-plus-full-text or OpenSearch covers it.
3. Is the answer authoritative structured data (prices, counts, status)? Then it's a tool call to a real database or API, full stop. Retrieval is the wrong primitive.
4. Is it large-scale conceptual recall over an unstructured corpus that won't fit in context? Now a vector database is the right tool, ideally inside a hybrid-search setup with a reranker.
5. Is it cross-modal or recommendation-style similarity? Vector database, no debate.
6. Is it agent long-term memory? Use a vector store for the genuinely fuzzy "recall a similar past episode" slice, and structured storage (a plain database) for everything with a key, a timestamp, or an entity you can index directly.

Notice that the vector database is the answer to two or three of six questions, not zero, not all of them. That ratio is the whole point. It's a specialized component, not a foundation.

Insights Most People Overlook

The vector database often hides a missing source of truth. Teams embed structured data, order records, user profiles, config, because embedding-everything was the pattern, then wonder why the agent returns stale or fuzzy answers about exact facts. The real fix isn't a better embedding model; it's recognizing that structured data should live in a structured store the agent queries directly. The vector DB became a graveyard for data that never should have been vectorized. Audit what you've embedded; a shocking fraction of it has a primary key and belongs in Postgres.

Chunking is the silent accuracy killer, and it's invisible in your metrics. When top-k retrieval drops the one chunk that held the answer, the agent doesn't error, it confidently answers from incomplete context. There's no exception, no log line, no alert. This failure is fundamentally a vector-pipeline artifact, and it's a major reason whole-document context (when it fits) often outperforms "better" RAG. If you can't fit the document, smaller-but-overlapping chunks and a reranker mitigate it, but you never fully escape it. Most teams have no measurement of how often this happens, which means they have no idea how much accuracy they're trading away.

Embeddings rot, and nobody budgets for it. When you change embedding models, and you will, because better ones ship constantly, every vector in your store is now in the wrong space and must be re-embedded. For a large corpus that's a real reindexing job with real cost and a migration window. This switching cost is a form of lock-in that the "just add a vector DB" tutorials never mention, and it's a genuine line item in agent CI/CD and versioning that operators discover the hard way.

"Vector database" and "the HNSW index in your Postgres" are not the same decision, and conflating them costs money. You can have vector search without a dedicated vector database. pgvector gives most agents all the approximate-nearest-neighbor capability they'll ever use, co-located with the structured data and keyword search they also need, with one system to operate instead of three. The dedicated managed vector database is justified by scale and specialized index performance, not by "we need vector search at all." Many teams pay for the former when they only needed the latter.

The agentic loop is slowly making retrieval quality a reasoning problem, not an index problem. As agents get better at composing multiple retrieval calls and verifying results, a mediocre retrieval layer with a smart agent on top can beat a beautiful retrieval pipeline feeding a one-shot prompt. That shifts where the engineering effort, and the budget, should go. Over-investing in the perfect vector index while under-investing in the agent's retrieval strategy is a common, expensive misallocation.

References

• Anthropic, Prompt Caching documentation
• Pinecone, Getting Started with Hybrid Search
• Andreessen Horowitz (a16z), AI infrastructure and the emerging stack