When is long-context cheaper than RAG?

Long-context wins when your corpus is small (a few thousand tokens) or your query volume is very low, so the saved engineering and infrastructure of RAG is not worth it. Above a few hundred thousand tokens of corpus at meaningful query volume, RAG is almost always far cheaper because you only pay for the small retrieved slice each call.

Why does RAG reduce per-query cost so much?

A long-context call pays for the entire corpus on every single request. RAG pays a one-time embedding cost to index the corpus, then each query only sends the top-k retrieved chunks — often a few thousand tokens instead of a million — so the per-query input cost can be 100x smaller.

Long-context prompts have high time-to-first-token because the model must process the whole context. RAG adds a small retrieval step (typically 20-80 ms for a vector search) but the generation call processes far fewer tokens, so end-to-end latency is usually lower for large corpora.

Does RAG hurt answer quality?

It can, if retrieval misses the relevant chunk. That is the precision tradeoff — long-context never misses because everything is in the prompt. Use RAG when your queries are answerable from a few focused passages; use long-context when the answer requires synthesizing the entire document.

Is this an exact cost figure?

No, it is a planning estimate. Real costs depend on cache hits, output tokens, reranking, and your vector database pricing. Use it to decide which architecture to prototype, then measure the real numbers.

RAG vs Long-Context: Cost & Latency Tradeoff

RAG vs long-context cost calculator

Long-context models like Gemini 1.5 Pro can hold a million tokens in a single prompt, which makes a tempting shortcut: skip the vector database and just paste your whole corpus into every request. That convenience has a cost — you pay for the entire corpus on every single query. This tool puts that number next to the cost of a RAG pipeline so you can decide with arithmetic instead of vibes.

How it works

Two strategies are priced over a 30-day month:

Long-context: every query sends the full corpus as input. Monthly cost is corpus_tokens × price_per_token × queries_per_day × 30. There is no indexing cost, but the per-query input is enormous.
RAG: a one-time embedding pass over the corpus (corpus_tokens × embed_price), then each query sends only the retrieved chunk (retrieved_tokens × query_price). Monthly cost is the amortized embedding plus retrieved_tokens × query_price × queries_per_day × 30.

Latency is modelled as a function of input size: long-context pays a per-token processing cost on the whole corpus, while RAG pays a fixed retrieval overhead plus processing of only the retrieved slice.

Tips and notes

The crossover point moves earlier (toward RAG) as corpus size and query volume grow — both multiply the long-context bill but not the RAG bill.
If your acceptable latency budget is tight and the corpus is large, RAG usually wins on latency too, not just cost.
Remember the hidden RAG costs this calculator does not include: the vector database, reranking, and the engineering time to build and maintain the pipeline. For a tiny corpus those can outweigh the token savings, which is exactly when long-context is the right call.