The Rule of Thumb and the Better Test

Put one document, or one small set of documents that is the same for everyone, straight into the context window and skip the retrieval pipeline. Build retrieval when the material is large, changes often or differs by who is asking. That rule of thumb is right.

The usual justification for it is that “a million-token context window still can’t hold 40 GB of SharePoint”. As a test that is the least useful, because plenty of real corpora do fit. A machine builder’s complete service manual set is maybe 200,000 tokens, and a regional insurer’s policy conditions for one product line 80,000. Both fit, and someone will tell you to put them into the prompt in full.

The better test asks about shape rather than size: is the front of the prompt byte-for-byte identical on every single query? If yes, long context often costs little. If no, it costs a lot, and no error message warns you, because you never hit the context window limit.

The one-line rule. Long context is affordable exactly when many queries reuse the same prompt prefix. If the prefix varies per user, per tenant or per hour, you pay full input price on every request, and retrieval wins on cost alone.

What a Long Prompt Actually Costs

A long-context design sends the document again with every question. Ten thousand support questions against a 200,000-token manual set come to two billion input tokens, not two hundred thousand. Anyone who skips that arithmetic does not notice that the supposedly simpler architecture is the more expensive one.

Prompt caching cuts the cost while the prefix stays the same

Every major provider now offers prompt caching, and only that makes long context pay rather than merely technically possible. The mechanics differ by provider. On Anthropic’s API a cache read costs a tenth of the normal input token price or less, depending on the model (as of October 2026), while writing the cache costs 1.25× (five-minute lifetime) or 2× (one-hour lifetime). With the five-minute cache, two requests against the same prefix already cost less than two uncached ones; with the one-hour cache that holds from the third request.

In the support example, the provider then no longer bills the 200,000-token manual set at full price 10,000 times but at a tenth of that or less, plus another write whenever the cache has expired. That cuts the cost by an order of magnitude, which is why “just put the handbook in the prompt” went from naïve to defensible.

The condition sits in the word prefix. Caching matches prefixes, and the cache key is derived from the exact bytes of the prompt from position zero up to the cache marker. If a single byte changes in that span, everything after it becomes invalid, and you pay full price again, with no error and no warning. Only the usage fields of the response show it, where cache_read_input_tokens drops to zero. A team that does not monitor those fields finds out from the invoice.

Per-user filtering destroys the cache unnoticed

This failure nearly always surfaces after the fact. The prototype puts all documents into the prompt, caching hits, and the numbers look good. Then, before production, someone asks whether a works-council member may see the same documents as a shift supervisor, and whether the Munich branch sees Hamburg’s contracts.

So the team filters the documents per user before assembling the prompt. The architecture gains only one filter step, but the cost then follows a different model. Every user now has a different prefix and therefore a separate cache entry, and you write each of those entries at 1.25× before you can read it at 0.1× or less. With a few hundred users asking a handful of questions each, most entries expire before a second request reads them. You have then paid the write premium and never collected the read discount.

A retrieval index has no such problem, because permissions there are metadata filters on the query, and the expensive part, embedding and indexing, happened once, offline, for everyone. This argument for RAG holds at any context-window size, because it is not about capacity.

Same docs for everyone handbook, product catalogue → One shared prefix cache hit from query two on → ≤ 10% of input price long context is affordable Filtered per user roles, branches, tenants → A prefix per user cache rarely warms up → Full price, every query retrieval wins on cost
The same architecture follows two cost models. Access control determines which one applies, so it cannot be added afterwards.

Effective Context Is Shorter Than the Advertised Window

The advertised window size says nothing about quality. A model with a 1M-token window does not reason equally well across the whole million.

Three papers show this. Liu et al.’s Lost in the Middle (2023) showed that accuracy is highest when the relevant passage sits near the start or the end of the input and drops measurably when it sits in the middle. NVIDIA’s RULER benchmark added multi-hop tracing and aggregation tasks and found that only about half of the tested models keep a usable quality at 32K tokens, although all of them advertise at least that length. Adobe Research’s NoLiMa (ICML 2025) is the sharpest of the three: it strips the literal word overlap out of needle-in-a-haystack tests, so the model has to infer the association rather than pattern-match the question’s own wording. Under that condition 11 of the 13 models tested drop below half their short-context baseline by 32K tokens.

Why the demo passes and production doesn’t

The NoLiMa condition explains a typical failure mode. In the demo, the presenter asks questions in the document’s own vocabulary, because they built the demo after reading it. Question and passage overlap in wording, the model finds the passage, and everything looks solved. In production, a service technician types “machine freezes when it tries to find zero” and the manual says “axis homing timeout.” Question and passage share no word and sit a hundred pages apart. The system that answered without error in the meeting starts inventing plausible-sounding answers.

A retrieval pipeline is not immune either. An embedding model that has never seen your domain vocabulary fails on the same query, which is why how you chunk and an additional keyword search matter so much. A long context window does not remove the retrieval problem, however. It moves retrieval into the model, where you cannot inspect it, log it or repair it with a metadata filter.

Test with other people’s questions. Build your evaluation set from questions by people who have not read the source documents, meaning real tickets, emails and phone notes. A long-context system that only passes questions phrased in the document’s own words has not been tested.

What a Prompt Can’t Do That an Index Can

Regardless of cost and quality, a prompt lacks three things, and most projects are decided by them.

Freshness. A prompt consists of whatever your code loaded at request time. If a price list changed this morning, either your assembly step detects that and re-reads the file, in which case you have written a small, unversioned ingestion pipeline, or you serve yesterday’s prices. An index has a defined update path and a defined moment from which new content is visible.

Provenance. When retrieval selects five passages, you know which five, which documents they come from and how retrieval scored them. You can show the source, log the selection and reconstruct months later why the system answered as it did. When you put 200 pages into a prompt, the model’s selection stays invisible. “It was in the context somewhere” is not an audit trail, and it will not satisfy anyone who has to assess a decision after the fact.

Deletion. Removing a document from the list your code builds the prompt from is trivial; removing it from a vector store is harder than it looks. That speaks for long context, but only while you run no index. Most systems end up with both, and then you have to solve the deletion problem anyway.

What I Build and What I Wouldn’t

What works in practice is the hybrid that retrieves generously and hands over generously. Pull thirty or fifty candidates instead of the traditional three, because the window can take it. But keep retrieving, so that the selection stays explicit, can be filtered by permission and can be inspected when something goes wrong. The larger context window lets you rank less precisely, but it does not let you skip that step. This is the same architecture I describe in RAG pipelines explained, only with a much wider funnel.

I skip retrieval where a human has already narrowed the scope to one artefact: a single contract under review, a tender document, an incident report or a code module. A retrieval pipeline there would index a corpus of exactly one document.

I would not use long context, even though it would work, when access rights depend on the user, when source data changes hourly or when you have to prove which source an answer came from. The prompt version is not simpler in those cases; it holds the same complexity in a place where you cannot see it.

All of this describes the cost and behaviour of today’s models at today’s prices, and both change. Access control, freshness and provenance, by contrast, are properties of your data and not of the model, and no context-window length changes that.