What is context refreshing?

Long-lived AI assistants cannot keep an entire history in the context window forever. Periodically they summarize the conversation and re-inject that compact summary so the assistant stays coherent. Each cycle costs output tokens to generate the summary and input tokens to re-inject it on subsequent turns.

Why does re-injection cost input tokens every turn?

Once a summary is re-injected, it sits in the prompt for every following request until the next refresh. So the re-injected context is billed as input tokens on each turn, not just once. This calculator models the per-refresh injection cost as a planning baseline.

How do I reduce context refresh cost?

Refresh less often, summarize more aggressively to shrink the re-injected size, use prompt caching so the stable summary is billed at a discount, and tier sessions so only active ones refresh.

Does prompt caching change the math?

Yes. If your provider supports prompt caching, the re-injected summary can be billed at a fraction of normal input cost on cache hits. Model the cached rate as a lower input price to see the savings.

What is a realistic refresh frequency?

It depends on usage. A daily personal assistant might refresh once per day; a busy support bot might refresh every few hours of active conversation. Match the frequency to how fast each session accumulates new context worth summarizing.

Context Refreshing Cost Calculator

Persistent AI assistants do not get memory for free. To stay coherent across days or weeks they periodically summarize and re-inject context — and that maintenance has an ongoing token cost that scales with your number of active sessions. This calculator makes that recurring spend visible.

How it works

Each refresh cycle has two cost components:

Generation cost — output tokens spent producing the summary of recent conversation.
Injection cost — input tokens spent re-inserting that summary into the prompt so the assistant carries it forward.

The tool computes:

cost per refresh = (gen_tokens × output_price + inject_tokens × input_price) / 1e6
monthly fleet cost = cost_per_refresh × refreshes_per_month × active_sessions

Refreshes per month derive from your chosen frequency. The result is broken into per-refresh, per-session-per-month, and whole-fleet totals.

Worked example

A support assistant that refreshes every 6 hours of active use, ~4 refreshes/day, 2,000 re-injection tokens and 600 generation tokens per refresh, at $1/1M input and $5/1M output, across 5,000 active sessions:

Cost per refresh: (600 × $5 + 2,000 × $1) / 1e6 = $0.005
Per session/month: $0.005 × 120 ≈ $0.60
Fleet/month: $0.60 × 5,000 = $3,000

Context maintenance alone is a $36k/year line item — worth optimizing.

Tips

Prompt caching is the biggest lever: model the cached input rate to see re-injection cost drop sharply.
Summarize harder. Halving the re-injected size halves the dominant input cost.
Tier dormant sessions so you only refresh ones in active use.
Pair with the LLM API Cost Calculator to fold this into total spend.