Retrieval Failures in AI Systems: What Goes Wrong and Why

7 October 2026

The answer was wrong, and the model got the blame.

Often the model did exactly what it was told. It answered from the context it was handed, and the context was wrong.

That is a retrieval failure. It needs a different fix.

What a retrieval failure is

A retrieval failure is any case where the retrieval layer hands the model the wrong context for the question. The output looks like a hallucination, and prompt tuning won't fix it.

The common shapes:

• Miss. The relevant chunk never comes back.
• Rank failure. It comes back, but below the cut-off for what the model sees.
• Near miss. A similar-looking chunk outranks the right one: last year's policy instead of this year's, the UK handbook instead of the US one.
• Stale. The right document, in an old version.
• Over-retrieval. The right chunk arrives buried in noise that dilutes or contradicts it.
• Leak. A chunk from a document the user's role does not cover.

Why retrieval failures are more common than you think

Retrieval failures hide well.

The answer reads fluently either way. The model doesn't flag that it was handed the wrong document; it writes a confident paragraph from whatever it got.

If you score only the final answer, a retrieval defect shows up as a "hallucination" and gets routed to whoever owns the prompt. Demos use questions the builders know the corpus can answer; users don't. Corpora change underneath the index. And every layer has settings (chunk size, overlap, embedding model, filters, reranking) that change what comes back.

Retrieval does not make a system immune. Stanford's "Hallucination-Free?" study (2024) found that 17–33% of answers from leading retrieval-based legal research tools contained a hallucination. Those are tools built specifically to anchor answers in sources.

The hidden cost of undetected retrieval errors

The direct cost is wrong answers. The hidden cost is effort aimed at the wrong layer: rewriting prompts, swapping models or adding guardrails for a problem that lives in the index. Then there is trust: a user who gets a confidently wrong answer either stops using your knowledge base or, worse, keeps using it.

Common causes of retrieval failures

How chunking strategy affects retrieval accuracy

Chunking decides what one unit of evidence is. Get it wrong and the right answer exists in your corpus, but never in one retrievable piece.

• Too small. A rule lands in one chunk and the condition that qualifies it lands in the next. "Employees may carry over unused leave" comes back; "with manager approval, up to the limit in the appendix" does not.
• Too large. One chunk covers several topics, its embedding blurs them together, and it matches nothing well.
• Structure lost. Tables flattened, headings stripped, so a chunk no longer says which product it belongs to.
• Bad parsing and boundaries. Sentences cut mid-way, scanned PDFs indexed without text, page headers repeated in every chunk.

Embedding misconfigurations and their impact

• Model mismatch. Queries embedded with a different model, or model version, from the one that built the index. The vectors are not comparable, and similarity scores turn into noise.
• Domain mismatch. General-purpose embeddings can blur internal jargon, product codes and acronyms. Exact identifiers often need keyword or hybrid retrieval.
• Silent truncation. Text longer than the embedding model's input limit is cut before embedding, so the end of a long chunk is invisible to search.

Filters, freshness and permissions

• Filters. A region or product filter that excludes the right document, or a date filter that was never applied.
• Freshness. Documents updated in the source system but not re-indexed; deleted documents still sitting in the index.
• Permissions. Vector search bypassing row-level security, so a junior user's question returns a chunk only senior users should see. That is sensitive information disclosure (OWASP LLM02), with LLM08 covering the vector layer. Our pass bar is zero leaks.

How to detect retrieval failures before they reach production

Key signals that indicate a retrieval problem

• Answers that cite the wrong document or an old version.
• Good answers to the builders' phrasing; bad ones to paraphrases.
• "I couldn't find that" when the document plainly exists.
• Quality that differs sharply by document type: tables, scans, long manuals.
• Quality that drops after a re-index or a chunking or embedding change.
• Prompt changes that don't move the failure rate.

What a rigorous retrieval evaluation looks like

• A test set where each question is labelled with the documents or chunks that should come back, not only the expected answer.
• Retrieval scored on its own: recall (did the relevant items come back), precision (how much of what came back is relevant), MRR (how high the first relevant item ranked) and NDCG (how good the whole ranking is).
• Latency measured on the same calls and reported as p50, p95 and p99.
• Results sliced by document type, question type, role and language. Averages hide disasters in small slices.
• Paraphrased and hostile versions of the same question.
• The same queries issued under each role, with written authorisation and a test account per role.
• Grounding scored separately afterwards, so you know which layer failed.

Fixing retrieval failures: from diagnosis to remediation

Diagnose first: which shape of failure, and which cause. Then fix in the layer that broke.

A miss on paraphrases points at embeddings or a case for hybrid retrieval. A rank failure points at reranking or how many results you pass on. A near miss between versions points at metadata, filters and freshness. A leak is fixed by enforcing permissions at retrieval time, not in the prompt. After each change, re-run the same test set against your baseline.

When to re-index vs when to re-embed

Rule of thumb: re-index when content or chunking changes; re-embed when the model producing the vectors changes. Never mix vectors from two embedding models in one index.

Why independent assessment catches what internal teams miss

The team that built the index chose the chunking and wrote the test questions, so the questions tend to fit the chunking. An outside assessor writes questions the way users ask them and scores retrieval apart from the answer.

At breaklight that is Retrieval Quality, followed by Answer Grounding so the two defects are never confused. Classic, hybrid and re-ranked retrieval are covered by the core; knowledge-graph and GraphRAG systems use adapted tests. Where a managed service hides the chunking and embedding layer, we can still measure outcomes, but root-cause diagnosis may be limited.

Building a retrieval-resilient AI application

• Treat the index as code: version the chunking rules, the embedding model and each index build.
• Keep a labelled retrieval test set and run it on every change to chunking, embeddings, filters, reranking or the corpus.
• Log retrieved chunk IDs and ranks with every answer, so a production failure can be traced to its layer.
• Enforce permissions at retrieval, not in the prompt.
• Know when each source last synced.
• Turn every retrieval failure in production into a test case.

When the model gets the blame, check what it was given first. Score only the final sentence and you will misdiagnose retrieval failures as hallucinations.

If you can't yet tell whether your wrong answers start in retrieval or in the model, that is the first thing an assessment settles. breaklight's AI Enabled Retrieval Accuracy assessment scores retrieval and grounding separately against an agreed test set, then hands you findings and a remediation roadmap in priority order. The method is in the breaklight whitepaper.