Every modern large language model — GPT, Claude, Gemini, Llama — is built on one idea introduced in the 2017 paper “Attention Is All You Need.” That idea is the attention mechanism. It sounds abstract, but the intuition behind it is surprisingly approachable, and understanding it demystifies how these models read and relate words.
The problem attention solves
Consider the sentence: “The trophy did not fit in the suitcase because it was too big.” What does “it” refer to — the trophy or the suitcase? A human knows it is the trophy, because only the trophy being big makes sense. To answer correctly, a model must look back across the sentence and weigh how relevant each earlier word is to the word it is currently processing. Older models processed text strictly in sequence and struggled to connect words that were far apart. Attention solves this by letting every word look at every other word directly.
Queries, keys, and values
Attention works through three vectors that the model computes for each word. The query represents what the current word is looking for. The key represents what each word has to offer. The value is the actual information each word carries. To process a word, the model compares its query against the keys of all words to produce relevance scores, then uses those scores to take a weighted average of the values. Words that score high contribute more. The result is a new representation of the word that is informed by exactly the context that matters. It behaves like a soft, learned lookup table where relevance is computed from meaning rather than position.
Self-attention in practice
When this happens within a single sequence — every word attending to every other word in the same text — it is called self-attention. Each word ends up with a representation enriched by the most relevant surrounding words. This is how the model figures out that “it” links to “trophy,” or that an adjective modifies a particular noun several words away. Because every position is compared with every other in one operation, context flows freely across the whole passage.
Multi-head attention
A single attention operation can only emphasise one pattern of relationships at a time. So transformers run several in parallel, called heads. One head might track grammatical subject-verb links, another might follow long-range references, another might focus on nearby words. Their outputs are combined, giving the model a rich, multi-faceted view of how the words relate. This is multi-head attention, and it is a major reason transformers capture language so well.
Why it changed everything
Before attention-only models, sequence models processed text one step at a time, which was slow and hard to parallelise. By relying on attention, transformers process all positions simultaneously, training dramatically faster on GPUs. That efficiency is what made it feasible to scale models to billions of parameters and trillions of tokens. The attention mechanism did not just improve quality — it unlocked the scale that produced the AI assistants we use today.