The two phases of a model’s life
Every machine learning model lives through two distinct phases. Training is the learning phase: the model is shown enormous amounts of data and its internal parameters are adjusted, over and over, until it captures patterns in that data. Inference is the doing phase: the trained model takes a new input it has never seen and produces an output. The clearest analogy is education — training is the years of study that build knowledge, and inference is answering a single question on the spot. You train a model once (or occasionally), but you run inference every single time someone uses it.
What happens during inference
When you send a prompt to an LLM, inference is the computation that turns your text into a response. The model converts your words into tokens, runs those tokens through its billions of parameters, and predicts the next token, then the next, until the answer is complete. Crucially, this is real work done fresh for your specific input — there is no lookup table of pre-written answers. That is why the same model can respond to infinitely many prompts, and why every response consumes real compute.
Why inference costs money
Because inference runs the full model on expensive GPU hardware for each request, it has a real per-use cost that traditional software does not. This is the heart of why AI providers bill by the token rather than a flat fee: a longer prompt and a longer answer literally require more computation. Over a product’s life, inference usually dominates total AI spend — training happens once, but inference happens millions of times — which is why so much engineering effort goes into making it cheaper.
Why latency matters
Inference also takes time, and that delay — latency — shapes how an application feels. A model generates text token by token, so larger models that do more computation per token respond more slowly. For a batch job this barely matters, but for a chatbot or a coding assistant, a sluggish response ruins the experience. Product teams therefore balance model size against speed, often choosing a smaller, faster model when responsiveness matters more than peak quality.
How inference is optimised
Engineers have a toolkit for making inference faster and cheaper without retraining. Quantisation stores the model’s numbers at lower precision, shrinking it and speeding it up with minimal quality loss. Batching processes many requests at once to use the GPU efficiently. Caching reuses computation for repeated context. Distillation trains a small model to mimic a large one, trading a little quality for big gains in speed and cost. Combined, these techniques let providers serve powerful models to millions of users affordably — which is what makes everyday AI possible at all.