Stop picking by reputation
The right LLM is not the most famous or the most powerful — it is the one that meets your task’s quality bar at the lowest cost and acceptable latency, within your privacy constraints. Teams that pick by hype overpay and ship slow products; teams that pick by a decision framework match each workload to the cheapest model that clears it. This guide gives you the axes that actually matter and how to weigh them, so the choice becomes an engineering decision rather than a brand preference.
The axes that actually matter
Capability and reasoning quality. Frontier models excel at multi-step reasoning, nuanced instructions, and ambiguous tasks; smaller models handle classification, extraction, and short generation just as well. Test candidates on your real prompts with a small evaluation set — published benchmarks rarely predict performance on your specific workload.
Context length. This is the token budget for prompt plus output. Long-document analysis, large codebases, and lengthy conversations demand big windows; short, transactional tasks do not. Do not pay for a million-token window to summarise tweets.
Latency. Interactive features (chat, autocomplete) need fast first-token times; batch jobs (overnight summarisation) can tolerate slow, cheaper models. Streaming hides latency for user-facing flows but does nothing for total throughput.
Cost per token. Prices vary by 10–50x within and across families, and it compounds at scale. Read input and output pricing separately, because output is usually several times more expensive.
Multimodal support. If you need to read images, parse documents with layout, or handle audio, you need a model built for it — text-only models simply cannot.
Privacy and data residency. For regulated data, where processing happens and whether prompts are retained can outweigh every other factor. Look for zero-retention modes, regional hosting, or self-hosted open models.
Fine-tuning and customisation. If you have proprietary data and a quality or cost ceiling prompting cannot break, fine-tuning availability matters — but treat it as a later optimisation, not a starting requirement.
A workable decision process
Start by writing down the task, the quality bar, the latency budget, and any hard privacy constraints — these eliminate most options immediately. Build a tiny evaluation set of ten to thirty real inputs with known-good outputs, and run two or three candidate models against it; this beats any leaderboard. Default to the smallest model that clears the bar, and consider a routing pattern where easy requests hit a cheap model and only hard ones escalate. Crucially, abstract the model behind a thin interface so the provider is a config value, not a dependency baked through your code. The landscape shifts every few months, and the teams that win are the ones who can re-evaluate and switch cheaply rather than the ones who guessed right once.