The real cost of long context: a worked example at 40,000 calls a day

Large context windows changed how teams prototype. Why build a retrieval pipeline when the whole product manual fits in one request? For a demo, that reasoning is fine. For a feature with real traffic, the arithmetic deserves ten minutes before anyone commits to it.

Here is that arithmetic for a support assistant answering questions about a 300-page manual.

The setup

Two designs, same model, same questions:

• Stuff everything. Send the full manual, about 120,000 tokens, plus the question with every request.
• Retrieve first. Embed the manual in chunks, fetch the top eight relevant ones, and send roughly 6,000 tokens.

For prices we use a common shape rather than any one vendor: $3 per million input tokens and $15 per million output tokens. Answers average 300 tokens either way.

def cost_per_call(input_tokens, output_tokens=300, in_price=3.0, out_price=15.0):
    return input_tokens / 1e6 * in_price + output_tokens / 1e6 * out_price

stuffed = cost_per_call(120_000)   # 0.3645
retrieved = cost_per_call(6_000)   # 0.0225

The bill

At 40,000 calls a day, a modest number for a support widget on a busy product:

That is a 16x difference, and output tokens barely register. Input is the whole story. A second detail people miss: retrieval itself is not free, but embedding the question and searching an index usually costs fractions of a cent per query and a few tens of milliseconds.

The wait

Cost is the line finance sees. Latency is the one users feel. Before the model writes a single token it has to process the input, a step usually called prefill. Prefill throughput varies widely by model and hardware, so take this as an illustration, assuming 20,000 tokens per second:

• 120,000 tokens: about 6 seconds before the first word appears.
• 6,000 tokens: about 0.3 seconds.

Six seconds of a blank chat bubble is long enough for many users to assume the feature is broken. Streaming does not help here; there is nothing to stream yet.

The accuracy question

The intuition behind stuffing is that more context means better answers. In our tests the opposite showed up on specific question types. When the relevant paragraph sat in the middle of a 120,000-token prompt, the model was noticeably more likely to answer from a nearby but wrong section, especially when the manual had near-duplicate pages for old and new versions of a feature.

Retrieval has its own failure: if the right chunk is not in the top eight, the model never sees it. The difference is that you can measure retrieval recall directly and fix it. Diagnosing why a model ignored a paragraph it was given is much harder.

When long context is the right call

Stuffing is not always wrong. It earns its cost when:

1. Traffic is low, say a few hundred calls a day for an internal tool.
2. The task needs the whole document, like summarizing a contract or comparing two versions.
3. The corpus changes hourly and an index would always be stale.
4. You are still learning what users ask, and a retrieval pipeline would be premature.

For a high-traffic question-answering feature, none of those usually hold. Build the retrieval step, measure recall, and keep the long context window for the jobs that truly need every page.