On this page 12 sections
Fitting a document into the context window is not the same as the model using all of it. Accuracy on facts placed in the middle of a long input is measurably worse than at the beginning or end, cost scales with everything you send, and latency follows. For long documents, retrieve the relevant sections rather than pasting the whole thing - the exception is a task that genuinely needs global structure, and those are rarer than they look.
“The window is large enough” answers a capacity question. It does not answer whether the answer will be right.
Key takeaways
- Position matters. Facts in the middle of a long input are recalled less reliably.
- You pay for every token you send, on every request, forever.
- Filling the window with loosely relevant text makes answers worse, not more informed.
- Retrieval plus reranking beats pasting the document for almost all extraction and Q&A.
- The genuine exception is a task needing the whole structure at once, and it needs a different design.
Who this applies to
You are working with documents longer than a few pages - contracts, reports, manuals, transcripts - and deciding between putting them in the prompt and building retrieval.
Three reasons the whole document underperforms
1. Position affects recall
Models do not attend uniformly across a long input. Information at the start and end is recalled more reliably than information in the middle, and the effect gets more pronounced as inputs get longer.
For a 200-page contract, a clause on page 90 is in the least reliable region. The model will usually find it; usually is not a specification when the question is whether a liability cap exists.
This is a property of how these models process long sequences rather than a bug that gets patched, so design around it rather than waiting for it to be fixed.
2. Irrelevant context degrades answers
Adding text that does not bear on the question makes answers worse, not merely more expensive. Loosely related passages compete for attention with the correct one, and the model may blend them.
This is why reranking often improves quality while reducing cost - see cutting LLM costs without degrading quality. Fewer, better passages beat more, weaker ones.
3. Cost and latency scale with input
You pay for every token on every request. Pasting 150,000 tokens of contract to answer a question about one clause costs that much each time somebody asks. Time to first token rises with input length too.
Retrieval turns a per-request cost into a one-off indexing cost. That difference compounds with volume.
What to do instead
Chunk with structure, not by character count. Split on the document’s own boundaries - sections, clauses, headings - rather than every 1,000 characters. A clause cut in half retrieves badly and reads worse.
Carry the hierarchy on each chunk. A chunk from “Section 7.3, Limitation of Liability” should record that path in its metadata and ideally in its text. Otherwise a retrieved paragraph loses the context that tells you what it governs.
Overlap slightly, so a fact spanning a boundary is not lost.
Retrieve, rerank, then send few. Twenty candidates, reranked, top three or five to the model.
Cite the section. For documents, the answer should say where it came from, so a person can check. That is also the mechanism that catches a confidently wrong retrieval.
When you genuinely need the whole thing
Retrieval is not always right. Three cases where it fails and long context is the correct tool.
Global questions. “Summarise this document” or “what are the main obligations” cannot be answered from three retrieved chunks, because the answer is a property of the whole.
Cross-references that span the document. “Does anything here contradict clause 4.2” requires seeing everything, and retrieval will only find what matches the query terms.
Ordering and narrative. “What happened, in sequence” from a long transcript.
For these, the workable patterns are hierarchical summarisation - summarise sections, then summarise the summaries - or a map-reduce pass over the document, or genuinely sending the whole thing while accepting the cost and the middle-of-document weakness.
The practical distinction: if the answer lives in a specific part of the document, retrieve. If the answer is a property of the whole document, you need a pass over the whole document. Most business questions about contracts and reports are the first kind, and most systems built for them use the second.
The hybrid worth knowing
For long documents with a stable structure, a two-stage approach outperforms both.
Stage one: retrieve or navigate to the relevant section using the document’s structure - a table of contents index, or a retrieval pass over section summaries rather than over raw chunks.
Stage two: send that entire section, not three fragments of it, to answer the question.
This keeps local coherence, which pure chunk retrieval loses, while avoiding the cost and the position problem of the whole document. For contracts, manuals and structured reports it is usually the best of the available designs, and it is under-used because it needs an indexing step that plain chunking does not.
What changes at corpus scale
The advice above assumes one long document. With thousands of them, two additional problems appear that single-document thinking misses.
Retrieval has to find the right document before the right section. With ten documents, any retrieval finds the right one. With ten thousand, near-duplicates compete - three versions of a contract, two of which are superseded. The failure is not a missing answer, it is a confidently correct answer from the wrong version.
The defence is metadata rather than better embeddings: version, effective date, status and entity on every chunk, filtered in the query. A superseded document should not be a candidate, and no amount of semantic similarity determines that - it is a business fact you have to carry.
Reindexing becomes a scheduling problem. One document reindexes instantly. A large corpus takes long enough that you need incremental updates driven by change events rather than a full nightly rebuild, and you need to know how stale the index currently is.
Both point the same way: at scale, the hard part of a document system stops being retrieval quality and becomes data management - knowing which version is current, which documents are superseded, and what has changed since the last index. That work is unglamorous, it is where these projects actually overrun, and it is usually scoped as an afterthought.
Testing whether it works
The failure here is quiet: the model produces a plausible answer from the wrong part of the document.
So test retrieval separately from generation. For each case in your evaluation set, record which section should have been retrieved, and measure how often it was. If the correct section is not retrieved, the answer is wrong regardless of how well the model writes.
That single metric - retrieval hit rate - is the most useful diagnostic in any document system, and it is frequently not measured because end-to-end answers look acceptable. See building an eval set from real tickets; the method applies unchanged to documents.
Stratify by where in the document the answer lives. If accuracy is materially worse for answers in the middle, that is the position effect showing up in your own system.
What we default to
Structure-aware chunking with hierarchy carried in metadata, retrieval with reranking, section-level retrieval for structured documents, and retrieval hit rate measured as its own metric.
The recommendation clients most often resist is not pasting the document, because long context windows are marketed as removing the need for retrieval and the simplicity is genuinely appealing. It does work in testing, on documents where the answer happens to sit near the start, and the cost only becomes visible at volume.
The argument that lands: ask what happens when the question is about page 90 of 200, and then ask what the same request costs multiplied by your monthly volume. Both answers point the same way.
Where we have been wrong: we defaulted to chunk retrieval for a long time on documents that needed whole sections, and the symptom was answers that were technically supported by the retrieved fragment and misleading without the surrounding clause. Section-level retrieval fixed it. If a system is producing answers that are individually defensible and collectively wrong, that is the shape of the problem.
When neither is the answer
When the document should be structured data. If you are repeatedly extracting the same fields, extract once into a database and query it. See document extraction.
When a person reads it once. Do not build a pipeline for a document nobody will ask about twice.
When the corpus is small and stable. Ten documents that rarely change may not need retrieval infrastructure - though they probably still need section-level chunking.
Frequently asked questions
Does a bigger context window remove the need for retrieval?
No. It removes the capacity constraint and leaves the position, cost and irrelevant-context problems. Those are the ones that affect answer quality.
How large should chunks be?
Follow the document’s structure rather than a token count. For prose, a section or a few paragraphs. For contracts, a clause. Coherence matters more than uniformity.
Should we summarise documents at index time?
For structured documents, section summaries make an excellent retrieval layer - you match against summaries and send the full section. It costs one indexing pass and improves both precision and coherence.
How do we handle tables in long documents?
Poorly, if you chunk them as text. Extract tables separately and preserve their structure, or keep each table whole in one chunk with its caption and heading attached.
What is the single most useful thing to measure?
Retrieval hit rate: how often the section containing the answer was actually retrieved. It isolates the failure that end-to-end scoring hides.
Next step
If your document system pastes whole files into the prompt, the cost and the middle-of-document weakness are both worth measuring before volume grows. The document processing engagement builds structure-aware retrieval with hit rate measured separately.
Related: Document extraction: what to verify · Cutting LLM costs without degrading quality · RAG permissions · Document processing