The Answer, and the Question People Get Wrong
The short version: put one document — or one small, stable set of documents — in the context window and skip retrieval entirely. Build a retrieval pipeline when the material is large, changes often, or differs by who is asking. That rule of thumb is right, and you can stop reading here if you only wanted the rule.
What’s worth another eight minutes is why it’s right, because the usual justification — “a million-token context window still can’t hold 40 GB of SharePoint” — is the least useful test. Plenty of real corpora do fit. A machine builder’s complete service manual set is maybe 300,000 tokens. A regional insurer’s policy conditions for one product line: 80,000. Both fit comfortably, and someone will tell you to just paste them in.
The better test is a question about shape, not size: is the front of the prompt byte-for-byte identical on every single query? If yes, long context is often startlingly cheap. If no, it is startlingly expensive — and nothing about the context window limit will warn you, because you never hit it.
What a Long Prompt Actually Costs
A long-context design does not send the document once. It sends it again on every question. Ten thousand support questions against a 200,000-token manual set is two billion input tokens, not two hundred thousand. That is the arithmetic people skip, and it is the reason a “simpler” architecture can quietly become the more expensive one. If you want the full picture of where money goes in an AI build, that is a separate discussion — here I only care about the per-query line item.
Prompt caching changes the math — until it doesn’t
Every major provider now offers prompt caching, and this is what genuinely made long context viable rather than merely possible. The mechanics on Anthropic’s API are representative: a cache read costs roughly a tenth of the normal input token price, while writing the cache costs 1.25× (five-minute lifetime) or 2× (one-hour lifetime). Two requests against the same cached prefix already beat sending it twice uncached.
Applied to the support example, the 200,000-token manual set stops being billed at full price 10,000 times and starts being billed at roughly a tenth of that, plus a periodic re-write when the cache expires. That is a real change in kind, not a discount. It is why “just put the handbook in the prompt” went from naïve to defensible.
The catch is in one word: prefix. Caching is a prefix match. The cache key is derived from the exact bytes of the prompt from position zero up to the cache marker. Change one byte anywhere in that span and everything after it is invalidated — you are back to full price with no error, no warning, and no log line saying so. You find out from the invoice.
Per-user filtering is what quietly kills the cache
Here is the failure I see most often, and it is nearly always discovered after the fact. The prototype pastes all documents into the prompt, caching works beautifully, the numbers look great. Then someone asks the obvious production question: should a works-council member see the same documents as a shift supervisor? Should the Munich branch see Hamburg’s contracts?
So the team filters the document set per user before assembling the prompt. Architecturally that feels like a small change — same design, one filter step. Economically it is a different system. Every user now has a different prefix, so every user gets their own cache entry, and each of those entries has to be written at 1.25× before it can ever be read at 0.1×. With a few hundred users asking a handful of questions each, most entries expire before they are read twice. You have paid the write premium and never collected the read discount.
A retrieval index has no such problem: permissions are metadata filters on the query, and the expensive part — embedding and indexing — happened once, offline, for everyone. This is the argument for RAG that survives every context-window increase, because it is not about capacity at all.
Effective Context Is Shorter Than the Number on the Box
Cost aside, there is a quality question, and the marketing number is not the answer to it. A model advertised with a 1M-token window does not reason equally well across all million.
Three results are worth knowing by name. Liu et al.’s Lost in the Middle (2023) showed that accuracy is highest when the relevant passage sits near the start or the end of the input and drops measurably when it is buried in the middle. NVIDIA’s RULER benchmark added multi-hop tracing and aggregation tasks and found that most models fall below a usable quality threshold well before their advertised length. Adobe Research’s NoLiMa (ICML 2025) is the sharpest of the three: it strips the literal word overlap out of needle-in-a-haystack tests, so the model has to infer the association rather than pattern-match the question’s own wording — and under that condition many models drop below half their short-context baseline by 32K tokens.
Why the demo passes and production doesn’t
That NoLiMa detail explains a failure mode I have watched play out more than once. In the demo, whoever is presenting asks questions using the document’s own vocabulary — they wrote the demo after reading the document. Retrieval-by-lexical-overlap is easy, the model finds the passage, everything looks solved. In production, a service technician types “machine freezes when it tries to find zero” and the manual says “axis homing timeout.” No shared words, no anchor, a hundred pages of distance. The long-context system that scored perfectly in the meeting starts inventing plausible answers.
This is not an argument that retrieval is immune — an embedding model that has never seen your domain vocabulary fails on exactly the same query, which is why how you chunk and how you handle hybrid keyword search matter so much. The point is narrower: a long context window does not remove the retrieval problem. It moves it inside the model, where you cannot inspect it, cannot log it, and cannot fix it with a metadata filter.
What a Prompt Can’t Do That an Index Can
Three practical gaps remain regardless of cost and quality, and they are the ones that decide most real projects.
Freshness. A prompt is assembled from whatever your code fetched at request time. If a price list changed this morning, either your assembly step knows that and re-reads the file — at which point you have written a small, unversioned ingestion pipeline without admitting it — or you serve yesterday’s prices. An index has a defined update path and a defined moment when new content becomes visible.
Provenance. When retrieval selects five passages, you know which five, from which documents, at which scores. You can show the citation, log the decision, and reconstruct later why the system answered as it did. When you paste 200 pages into a prompt, the model’s selection is invisible; “it was in the context somewhere” is not an audit trail, and it will not satisfy anyone reviewing a decision months later.
Deletion. Removing a document from a prompt-assembly list is trivial; removing it from a vector store is genuinely harder than it looks. That asymmetry is real and it is one honest point in favour of long context — but it only holds while you have no index at all. Most systems end up with both, and then the deletion problem is yours regardless.
What I Actually Build — and What I Wouldn’t
In practice the interesting systems are not on either end of this axis. The pattern that works: retrieve aggressively, then be generous. Pull thirty or fifty candidate passages instead of the traditional three, because the window can take it — but still retrieve, so that selection stays explicit, filterable by permission, and inspectable when it goes wrong. The context window bought you room to be less precise at the ranking stage; it did not buy you the right to skip the stage. That is the same architecture I describe in RAG pipelines explained, just with a much wider funnel than was affordable two years ago.
Where I would genuinely skip retrieval: a single contract under review, one tender document, one incident report, one codebase module. Anything where a human has already narrowed the scope to one artefact. Building a retrieval pipeline there is engineering theatre — you are indexing a corpus of one.
Where I would not use long context, even though it would work: anything with per-user access control, anything with hourly-changing source data, and anything where you will be asked to prove which source produced an answer. In those cases the pasted-prompt version is not simpler, it is the same complexity relocated somewhere you cannot see it.
And the honest caveat on all of this: these are cost and behaviour characteristics of today’s models and today’s pricing. Both move. What does not move is the structural point — access control, freshness, and provenance are properties of your data, not of the model, and no context window length changes them.
Where Tippel Fits
Most of the time this decision is made implicitly, by whoever built the prototype, and only revisited when the first monthly bill or the first works-council question arrives. It is much cheaper to make it deliberately, with your real documents, your real access rules, and questions written by the people who will actually ask them.
That is what the AI Readiness Check does: one use case, your data, a fixed window, and an honest verdict at the end — including the verdict that the simplest version is enough and you do not need a pipeline. If you would rather just talk the trade-off through first, get in touch.