128k context: why it doesn't replace retrieving well
We dropped the entire manual into the prompt because it finally fit. Accuracy went down and the bill went up.
The idea was seductive: we could drop the entire manual into the prompt because it finally fit. The context limit had grown to 128k tokens. Every page was now reachable. We don't need retrieval anymore.
The experiment broke the theory. Accuracy went down. The bill went up. Asking the model to find the relevant section inside a wall of text is less reliable than retrieving the relevant section first.
With a lot of irrelevant context, the model gets distracted: you get correct answers about the wrong section. It confidently explains the procedure from chapter seven when you asked about chapter three. Both answers are plausible, so the model picks one and sounds certain.
Retrieving five good chunks still beat sending four hundred pages, on quality and on cost. A focused retrieval step finds the right section first. The model then reads something that's actually relevant. It answers correctly and more concisely.
Long context is an excellent safety net. It's a fallback for when retrieval misses something. For critical systems it's insurance. But it's not a retrieval strategy.
The seductive idea dies hard because long context works for small problems. It just doesn't scale to enterprise content volumes, where the signal-to-noise ratio collapses.