What is time-to-first-token?

Time-to-first-token (TTFT) is the delay between sending your request and receiving the first token of the response. It is dominated by the prefill phase, where the model processes your entire prompt, so longer prompts increase TTFT. With streaming, TTFT is what the user perceives as responsiveness.

Why compare streaming and non-streaming?

Non-streaming makes the user wait for the full response before seeing anything, so perceived latency equals TTFT plus the full generation time. Streaming shows tokens as they arrive, so the user sees output after just the TTFT. For long outputs the difference in perceived responsiveness is dramatic.

How accurate are the estimates?

They use representative published throughput benchmarks and are meant for planning and comparison, not SLAs. Real latency varies with provider load, region, batching, prompt caching, and network conditions. Treat the numbers as ballpark figures and measure your own production traffic.

Does prompt length affect latency?

Yes. The prefill phase processes every input token before generation begins, so a long prompt raises time-to-first-token. The estimator adds a per-input-token prefill cost on top of each model's base TTFT to reflect this.

LLM Latency Estimator

Estimate before you commit to a model

Cost is not the only axis when choosing an LLM — latency shapes the whole user experience. A model that is cheaper per token but twice as slow can make a chat feel sluggish. This estimator turns a model choice plus token counts into time-to-first-token and total latency, and shows how much streaming changes what the user actually feels.

How it works

Each model carries two benchmark figures: a base time-to-first-token and an output throughput in tokens per second. The estimator adds a small per-input-token prefill cost (longer prompts take longer to process before the first token) to the base TTFT, then divides expected output tokens by throughput to get generation time. It reports both, and contrasts the perceived latency of streaming (user waits only for TTFT) against non-streaming (user waits for the whole response).

Tips for using the numbers

Stream anything over a sentence or two. The perceived-latency gap grows linearly with output length — streaming keeps long answers feeling fast.
Shorten prompts to cut TTFT. Prefill scales with input tokens; trimming a bloated system prompt directly lowers the time before the first token.
Pick the faster model for interactive UX, the cheaper one for batch jobs where total throughput matters more than responsiveness.
These are planning estimates — always measure real production latency, which varies with provider load and region.