Retrieval Quality Is a Chunking Problem, Not a Model Problem

A retrieval feature that returns the wrong three chunks is indistinguishable, to the user, from a model that hallucinated. The chat answer is confidently wrong, the citation points at a paragraph that does not contain the answer, and the bug report says "the AI is making things up". So the first instinct is to reach for a better model — swap the embedding model, bump the dimensions, maybe try the reranker everyone is talking about this month.

We ran an OCR and document pipeline feeding a vector store of roughly 21,000 embedded chunks for in-app retrieval and chat. Over the life of that system, the embedding model changed once. The chunking strategy, the metadata schema, and the query path changed many times, and every meaningful jump in answer quality came from one of those three.

That is the pattern worth internalizing. Embedding models are broadly good at the thing they do: putting semantically similar short passages near each other. They are not good at compensating for a chunk that has been cut in half, a document whose tables were flattened into word soup by the extractor, or a filter that quietly excluded the only document containing the answer. Those are the failures you will actually hit.

Diagnose before you swap anything

You cannot fix retrieval you cannot measure, and "it feels better" is not measurement. Before touching any component, build a small evaluation set. Not a benchmark — thirty to fifty real questions, each paired with the chunk IDs that genuinely contain the answer.

Getting those pairs is manual work. Pull the questions from real usage logs if you have them, or from the people who asked for the feature if you don't. Then read the corpus and mark the answer spans yourself. It takes an afternoon and it is the highest-leverage afternoon in the project, because from then on every change produces a number.

Then split the metric in two, because the two halves fail for different reasons:

• :Recall@k — did the correct chunk appear anywhere in the top k you fetched? If this is low, the problem is upstream: extraction, chunking, metadata, or filtering.
• Precision at the prompt — of the chunks you passed to the model, how many were relevant? If recall is fine but answers are still bad, the problem is ranking, context ordering, or the prompt itself.

The decision rule we use: if recall@20 is below about 0.9, stop looking at models entirely. No embedding model or reranker recovers a chunk that never made the candidate set. Fix the pipeline. Only once recall@20 is high and recall@5 is much lower do you have a genuine ranking problem, and that is the one narrow case where a reranker or a different embedding model is the right tool.

Extraction quality sets the ceiling

Embeddings are computed on text. If the text is wrong, nothing downstream saves you. This is the least glamorous part of a retrieval system and the most common cause of unexplainable misses.

Things we have watched break real extraction:

• Two-column PDFs read across the columns, interleaving unrelated sentences.
• Tables flattened to a single line, so a row's values are separated from their headers by two hundred characters of other rows.
• Headers and footers repeated on every page, so the top-scoring chunks for many queries are boilerplate.
• Scanned pages where the OCR confidence was poor and nobody looked, producing plausible-looking nonsense that embeds fine and matches nothing.
• Superscripts and footnote markers glued into words.

The mechanism here is simple and skipped constantly: before you index, dump 20 random extracted documents to plain text files and read them. Not the JSON, not the chunk metadata — the actual text a model will see. You will find the problem in ten minutes. On construction plans in BuildAEC we treat extraction output as a reviewable artefact for exactly this reason, with per-item confidence surfaced to a human rather than assumed correct, which is the same argument we make about confidence scoring generally.

Chunking is where most recall is lost

The default advice — 512 tokens, 50 token overlap, split on whitespace — is a reasonable starting point and a terrible finishing point. It fails because it is blind to the shape of your documents.

Split on structure, not on length

A fixed-length splitter will happily cut a policy clause in half, leaving the condition in chunk 41 and the consequence in chunk 42. A query about the consequence retrieves a chunk that mentions it without the condition that governs it. The answer is then wrong in a way that looks like model failure.

Split on the document's own boundaries first — headings, clause numbers, list items, table rows — and only apply a length cap inside those units. If a semantic unit exceeds the cap, split it and repeat the parent heading into each part.

Give each chunk enough context to stand alone

A chunk is retrieved in isolation and read in isolation. "It must be submitted within 14 days" is useless without knowing what "it" is. The cheap fix is to prepend a breadcrumb of the document title and heading path to the text you embed:

{
  "chunk_id": "doc_8812:s3.2:p4",
  "embed_text": "Supplier Agreement 2024 > 3. Invoicing > 3.2 Disputes\n\nA disputed invoice must be submitted within 14 days of receipt...",
  "display_text": "A disputed invoice must be submitted within 14 days of receipt...",
  "doc_id": "doc_8812",
  "tenant_id": "t_4471",
  "doc_type": "contract",
  "effective_from": "2024-01-01",
  "page": 12,
  "prev_chunk": "doc_8812:s3.2:p3",
  "next_chunk": "doc_8812:s3.2:p5"
}

Two details in there matter more than they look. Separating embed_text from display_text lets you enrich what gets embedded without polluting what gets cited. And storing prev_chunk / next_chunk lets you expand a retrieved hit into its neighbors at query time — retrieve precisely, read generously. That single change fixed more truncated-answer complaints for us than any model swap.

The query side is half the system and gets a tenth of the attention

Users do not type well-formed passages. They type "what about the 14 day thing", or a part number, or a follow-up that only makes sense given the previous three turns. Embedding that string directly and hoping is not a strategy.

Four fixes, in the order we would apply them:

1. Rewrite follow-ups into standalone questions. In a multi-turn chat, resolve pronouns and implied subjects against the conversation before you embed. One cheap model call. Biggest single win on conversational retrieval.
2. Add lexical search alongside vectors. Embeddings are bad at exact tokens — SKUs, invoice numbers, error codes, surnames. A BM25 or Postgres full-text index catches those, and fusing the two candidate lists is far more robust than either alone. If your corpus is full of identifiers, this is not optional.
3. Filter on metadata, and check your filters. Tenant, document type, effective date. A filter bug is the quietest recall killer there is, because it looks like the answer simply is not in the corpus. Log the filter applied on every query.
4. Expand hits to neighbors before prompting. Retrieve top-k chunks, then pull adjacent chunks for the top few. Costs tokens, buys coherence.

Note that steps 1 and 2 are not model quality work at all. They are plumbing. That is the point.

When the embedding model genuinely is the problem

There are real cases, and it is worth naming them so this doesn't read as dogma.

• Non-English or multilingual corpora. Model choice matters a lot here. Building EzyEvite across seven languages taught us not to assume an English-tuned model generalizes.
• Heavily domain-specific vocabulary where general models place unrelated jargon terms close together — some medical, legal and scientific corpora.
• Very short chunks, like product titles or single table rows, where there is no surrounding prose for the model to work with.
• High recall@20, low recall@5. The right chunks are in the candidate set and something is ordering them badly. This is the reranker's actual job.

The tradeoff on changing models is worth stating plainly: it means re-embedding your entire corpus, and if you serve many tenants you need a migration plan that keeps old and new indexes queryable during the transition. We have done large per-tenant migrations with a reconciliation report gating every cutover, and the same discipline applies here. It is not a config change. That cost is the reason to exhaust the cheaper fixes first, not merely the order they happen to appear in.

An order of operations

When retrieval is bad and someone wants to change the model, this is the sequence we work through:

1. Build the 30–50 question eval set with known-good chunk IDs.
2. Measure recall@5 and recall@20 separately.
3. Read 20 extracted documents as plain text. Fix extraction.
4. Check chunk boundaries against document structure. Fix splitting.
5. Add heading breadcrumbs to embedded text and neighbor pointers to metadata.
6. Rewrite conversational queries into standalone ones.
7. Add lexical search and fuse the results.
8. Audit metadata filters with query-level logging.
9. Only now: try a reranker.
10. Only after that: try a different embedding model.

Steps 1 through 8 are ordinary software engineering — parsing, schemas, indexes, logging. Steps 9 and 10 are the ML-shaped ones, and they are last because they are the least likely to be the cause and the most expensive to change. This is the same instinct behind asking whether a feature needs an LLM at all: the interesting-looking component is rarely the one that is broken.

One more thing that only becomes obvious in production: retrieval quality decays. Documents get added, formats change, someone starts uploading scans instead of native PDFs, a new tenant's corpus looks nothing like the ones you tuned on. Run the eval set in CI against a fixed snapshot, and re-run it on real traffic samples monthly. Retrieval is not a feature you ship; it is a system you keep measuring. If you are building one and want a second pair of eyes on the pipeline before it hits real users, get in touch.