Context windows have expanded by orders of magnitude — from a few thousand tokens to hundreds of thousands and beyond. The marketing implication is that you can now put an entire corpus in the prompt and stop building retrieval systems. The research does not support that as cleanly as the numbers suggest.
Position affects recall
A well-replicated finding is that information placed in the middle of a very long context is retrieved less reliably than information at the beginning or the end. The effect varies by model and has improved, but the general shape persists: a long context is not uniformly attended to, and a fact buried in the middle of a large document may as well not be there for some queries.
Retrieval and long context solve different problems
Long context is excellent when a task genuinely requires reasoning across a whole document — following an argument, comparing sections, tracking something that changes. Retrieval is better when the answer lives in a small identifiable part of a large corpus, because it costs less, runs faster, and produces citations.
The two are complementary. Retrieve well, then give the model generous context around what was retrieved.
The cost is not linear in usefulness
Filling a very long context is expensive on every request and slows the response. A system that pastes in a hundred thousand tokens of documentation to answer a question whose answer is in one paragraph is paying a great deal for the convenience of not having built retrieval — and, on current evidence, is also more likely to miss the paragraph.