Most teams debugging a broken RAG pipeline start with the prompt. They should start with the vector database — or more precisely, with whichever of the many available vector databases is actually doing the retrieval work underneath it. A retrieval-augmented generation system is only as good as the passages it retrieves, and the layer that decides which passages surface — index type, chunking strategy, filtering logic, hybrid search — sits well below the model.

A 2026 benchmark of legal RAG systems put a number on this: swapping in a stronger embedding model lifted end-to-end answer correctness by 17.5 points and retrieval accuracy by 34 points, while the choice of LLM moved the needle far less. Retrieval, not generation, is where most RAG systems actually win or lose.

That puts vector databases at the center of the reliability conversation, not the periphery. The teams shipping RAG pipelines that hold up in production aren’t the ones with the fanciest prompt templates — they’re the ones who made six specific infrastructure decisions about their vector databases correctly before the first user query ever hit the system. Vector databases don’t get less important as a product matures; if anything, the decisions made early — index type, chunking approach, filtering strategy — become more expensive to unwind the longer a corpus grows on top of them.

Postgres or a dedicated vector database is a workload decision, not a preference

Reaching for pgvector because the team already runs Postgres is a reasonable default, not a compromise. Postgres extended with vector search handles tens of millions of embeddings comfortably, and it keeps vector data next to the relational data it describes — no second system to keep in sync, no separate backup strategy, no new operational surface for the on-call rotation to learn. For most RAG pipelines below the hundred-million-vector mark, that simplicity outweighs whatever a purpose-built option among dedicated vector databases offers on paper. Choosing between vector databases isn’t a one-time architectural decision made in a design doc and forgotten — it’s a tradeoff worth revisiting as the corpus and query volume both grow.

Throughput and tail latency pull in different directions

The tradeoff shows up clearly once a corpus grows large. A 50-million-vector benchmark comparing Postgres (extended with the pgvectorscale extension) against Qdrant found Postgres delivering roughly 11.4x higher throughput at 99% recall — 471.57 queries per second versus 41.47 — a gap wide enough that most teams evaluating vector databases can’t treat it as noise, while Qdrant held a clear edge on tail latency and index build time.

Neither result makes the other of these vector databases wrong. It means the decision has to be made against the actual workload: a batch-style RAG pipeline answering internal support tickets cares about throughput; a customer-facing chat product with strict p99 latency targets cares about the opposite. Benchmarks like this one are useful exactly because they force teams to compare vector databases on the metric that actually matters for their own traffic pattern, rather than on whichever number a vendor leads with.

Three signals say it’s time to move off Postgres

A single-node HNSW index in Postgres needs roughly 20-25KB of RAM per vector at typical embedding dimensions, and that number climbs fast once a corpus crosses tens of millions of rows — the first signal to watch is memory pressure that autovacuum and connection pooling tuning can no longer absorb. The second signal is write churn outpacing index rebuild time.

The third is a recall requirement that Postgres’s iterative scan settings can’t hit without unacceptable query latency. One of those three showing up is a tuning problem for the vector databases already in place. All three showing up together is a migration conversation — and it’s worth having that conversation deliberately, evaluating a shortlist of dedicated vector databases against the specific workload, rather than defaulting to whichever option is trending in a conference talk that quarter.

Chunking strategy moves recall more than model choice does

Every RAG pipeline starts with a decision that gets far less scrutiny than it deserves: how to cut source documents into retrievable pieces before they ever reach one of the vector databases doing the actual retrieval. Chunk too large and irrelevant text dilutes the embedding, pulling weakly related passages into the top-k results. Chunk too small and the passage loses the surrounding context an LLM needs to answer correctly, even when the vector databases involved technically retrieved the “right” match. Chunking strategy is easy to underestimate precisely because vector databases will happily index and return whatever they’re given — a bad chunk boundary doesn’t throw an error, it just quietly produces a worse answer.

Fixed-size chunking is still the right default for most corporations

For teams starting from zero, fixed-size chunking with a reasonable overlap window remains the sensible baseline — split text into consistent token or character windows, carry a small overlap between adjacent chunks so context doesn’t get severed at the boundary, and move on to the parts of the pipeline that matter more. A widely used implementation walkthrough documents exactly this pattern as the entry point most teams reach for before their vector databases have enough evaluation data to justify anything more elaborate,

Expensive chunking methods rarely earn back their computational cost

Semantic chunking, recursive chunking, LLM-based chunking, clustering-based chunking — the menu of increasingly sophisticated options keeps growing, and each one costs more compute and more latency to run at ingestion time. A systematic 2026 evaluation benchmarking eight chunking methods against multiple QA datasets found that the more expensive approaches did not produce proportional gains in retrieval quality over cheaper baselines like fixed-size chunking.

That doesn’t mean advanced chunking is never worth it — a corpus of long, structurally complex legal or technical documents can genuinely benefit from semantic boundaries. It means the decision should be justified by evaluation results specific to a team’s own corpus and documents, not by whichever technique reads best in a conference talk. Whatever chunking method a team lands on, the resulting chunks are only as useful as the vector databases indexing them are configured to retrieve — chunking and indexing are two halves of the same recall problem, and optimizing one without the other leaves performance on the table.

Retrieval failures, not generation failures, cause most RAG hallucinations

It’s tempting to blame a wrong answer on the LLM. The evidence increasingly points elsewhere — specifically, at how well the vector databases underneath a RAG pipeline are actually retrieving relevant passages in the first place.

Retrieval sets the ceiling on what the model can possibly get right

The Legal RAG Bench evaluation — built from 4,876 passages of Victorian criminal law and 100 hand-crafted expert questions — tested three embedding models and two frontier LLMs in a full factorial design specifically to separate the two effects. The finding was unambiguous: information retrieval — governed almost entirely by how well the underlying vector databases are configured — was the primary driver of end-to-end correctness and groundedness, with the choice of LLM producing a far more moderate effect.

The researchers went further, concluding that many errors that look like hallucinations in production legal RAG systems are actually triggered upstream by retrieval failures — the model wasn’t wrong, it was answering from the wrong passages. No LLM upgrade fixes a retrieval problem, and teams that keep swapping models while recall stays flat are optimizing the wrong layer of vector databases and application logic entirely.

Metadata filtering after retrieval silently destroys recall

A subtler failure mode shows up in how filters get applied across most production vector databases. Filtering by metadata — tenant ID, document type, date range — after an approximate nearest-neighbor search has already returned its top-k candidates can quietly cut the result set well below what the application expected, because the filter discards matches that never should have survived the ANN pass in the first place.

Newer pgvector releases address this directly: iterative index scans (hnsw.iterative_scan set to strict_order or relaxed_order) keep searching until the requested number of filtered results is actually satisfied, instead of returning a truncated set silently. Teams running older configurations of their vector databases without iterative scans enabled are often debugging a “bad embedding model” that is actually a filtering bug.

For teams building the ingestion and indexing side of this pipeline, the patterns used to move documents from source systems into embeddings and back out again borrow heavily from real-time data pipeline design — the same backpressure and ordering guarantees that matter for streaming analytics apply directly to keeping vector databases current as source documents change underneath them.

Hybrid search is no longer optional for production RAG

Semantic-only vector databases miss queries that depend on exact terms — product codes, legal citations, error messages, proper nouns that an embedding model has never learned to distinguish precisely. Pure keyword search misses queries phrased in natural language that share no vocabulary with the source text. Production RAG pipelines increasingly run both and merge the results, and the vector databases mature enough to support this out of the box have a real operational advantage over ones that only do approximate nearest-neighbor search.

Lexical and semantic search fail on different query types

The two retrieval methods are complementary precisely because they fail differently. A query like “the June incident where the payment retries doubled” needs semantic understanding to connect loosely related phrasing to the right passage. A query like “error code E4471” needs exact lexical matching that an embedding model may blur into something only approximately similar. Combining both, rather than picking one, closes gaps that neither method covers alone regardless of which vector databases sit underneath — a pattern covered in detail across nine advanced RAG techniques worth adopting once a pipeline moves past a proof of concept. 

Reciprocal rank fusion merges results without an apples-to-oranges score problem

The technical obstacle to hybrid search is that lexical search scores (BM25-style) and vector similarity scores live on completely different scales, so naively averaging them produces meaningless rankings. Reciprocal Rank Fusion sidesteps the problem by ignoring raw scores entirely and combining results based on each document’s rank position in each result list — a technique that works the same way regardless of which vector databases produced either ranked list. Postgres can implement this directly by pairing a tsvector/GIN full-text index with sparse vector support for lexical-style matching, then fusing the two ranked lists at query time — no separate search engine required for teams already committed to vector databases running on Postgres.

Index build time and write churn decide your re-embedding strategy

Among vector databases, an index built once and never revisited is a liability. Source documents change, embedding models get upgraded, and both events force a decision about how much of one of these vector databases has to be rebuilt versus updated incrementally. Teams that treat their vector databases as static infrastructure — configured once at launch and left alone — are the ones most surprised when recall quietly degrades six months later.

Re-embedding after a model upgrade is an operational event, not a config change

Swapping embedding models — moving to a newer version with better benchmark scores — means every existing vector already sitting in the vector databases was produced by a model that no longer matches new queries. Mixing embeddings from two different model versions in the same index degrades recall silently, because similarity scores between old and new vectors are not meaningful. That makes a model upgrade a full re-embedding job across the entire corpus of vector databases, not a drop-in swap: it’s fundamentally a batch operation that needs to be scheduled and monitored like any other data migration, distinct from the incremental streaming updates the same pipeline handles the rest of the time.

Incremental indexing beats full rebuilds for high-churn corpora

For corpora with frequent updates — support tickets, product catalogs, chat transcripts — rebuilding entire vector databases on every change is wasteful and slow. An event-driven approach that captures document changes as they happen and pushes only the affected chunks through re-embedding keeps the index current without the operational cost of periodic full rebuilds. The same event-driven architecture patterns used for other data-consistency problems apply cleanly here: a change-data-capture stream watching the source of truth, with idempotent consumers writing incremental updates into whichever of the vector databases sits downstream.

It’s also worth naming who — or what — actually runs those consumers. Incremental indexing jobs are frequently triggered by service accounts and background workers rather than human users, and those non-human identities need the same scoped, auditable access controls as any other automated process touching production data — a gap that shows up repeatedly wherever AI agents and pipeline workers are granted standing credentials to write directly into a vector database.

Production RAG needs the same observability discipline as any other system

A RAG pipeline that isn’t instrumented looks fine right up until users start noticing it isn’t. Recall and latency don’t degrade all at once — they drift, and the vector databases at the center of the pipeline rarely surface that drift on their own without deliberate monitoring built around them.

Recall and latency percentiles deserve dashboards, not spot checks

Query latency at p50 rarely tells the story a team needs about how their vector databases are actually performing. The queries that matter — the ones that make a RAG pipeline feel broken to a user — show up at p95 and p99, and they’re driven by different causes than the median: a cold cache, a poorly filtered query, an index segment that needs rebuilding.

Teams already running distributed tracing for other services can extend the same instrumentation to the retrieval layer rather than building a bespoke monitoring stack from scratch, using the same three pillars of observability — traces, metrics, and logs — applied to embedding calls, index queries, and reranking steps as a unified pipeline rather than isolated black boxes. Treating vector databases as just another instrumented service, rather than an opaque dependency, is what makes a p99 latency spike or a recall regression something a team can diagnose in minutes instead of days.

Embedding drift is invisible until users notice

Source content changes over time, user query patterns shift, and the embedding model’s understanding of “similar” was frozen at training time — none of which shows up as an alert from the vector databases themselves. None of that shows up as an error — it shows up as a slow decline in whether retrieved passages are actually relevant, which is much harder to catch without a recall benchmark run on a regular cadence against a held-out set of representative queries. Vector databases don’t fail loudly; they fail by quietly returning the fourth-best answer instead of the best one, and the only way to catch that early is to measure it deliberately rather than wait for a support ticket.

Getting the infrastructure right before the prompt

None of these six decisions about vector databases are visible to an end user, and none of them show up in a demo. They show up three months into production, when query volume climbs, the source corpus grows past whatever fit comfortably in memory during testing, and the gap between “works in the demo” and “works at scale” becomes impossible to ignore.

Teams that treat vector databases as a solved problem — pick one, ship it, move on — are the ones debugging mysterious recall drops during an incident. Teams that treat the choice of index, chunking strategy, filtering logic, and observability stack as first-class engineering decisions are the ones whose RAG pipelines still work the same way in month six as they did in week one.

Landskill’s engineering teams build and operate exactly this kind of production data infrastructure — including the vector databases underneath it — for clients moving RAG pipelines from prototype to scale. Get in touch to talk through what a vector database migration or a hybrid search rollout looks like for your specific corpus.