The transformer is the architecture behind every modern large language model — GPT, Claude, Gemini, Llama. Introduced in the 2017 paper “Attention Is All You Need,” it replaced the step-by-step recurrent models that came before with something faster and far better at understanding context. You do not need any mathematics to grasp how it works; you need four ideas.
Tokens: breaking text into pieces
A transformer never sees words or letters directly. Text is first split into
tokens — chunks that are often whole words but sometimes word fragments, like
token plus isation. Each token is mapped to a long list of numbers called an
embedding that represents its meaning. Words used in similar ways end up with
similar embeddings, so the model starts with a numeric sense of what each token
means before it has read any context.
Self-attention: words looking at words
The heart of the transformer is self-attention. For every token, the model asks: which other tokens in this sequence should I pay attention to in order to understand this one? Consider “The trophy didn’t fit in the suitcase because it was too big.” To resolve “it,” the model must attend strongly to “trophy” rather than “suitcase.” Attention computes exactly these weightings — for each token, a set of scores saying how relevant every other token is — and blends the attended tokens together into a new, context-aware representation.
Crucially, this happens for all tokens in parallel, not one at a time. That parallelism is why transformers train so much faster than the recurrent models they replaced, and why any token can directly influence any other no matter how far apart they sit.
Positional encoding: restoring word order
Because attention looks at every token simultaneously, the model has no inherent sense of order — “dog bites man” and “man bites dog” would be indistinguishable. Positional encoding fixes this by adding a position signal to each token’s embedding, so the model knows token three comes after token two. Modern models use refined schemes like rotary embeddings, but the purpose is the same: give the order-blind attention mechanism a sense of sequence.
Feed-forward layers and stacking
After attention has mixed information between tokens, a small feed-forward network processes each position on its own, transforming the gathered context into richer features. One attention step plus one feed-forward step makes a block, and the magic of transformers is stacking — repeating this block dozens or even hundreds of times. Early blocks capture surface patterns like grammar; deeper blocks build abstract meaning and reasoning. By the top of the stack, each token’s representation encodes a deep understanding of its role in the whole passage.
A language model adds one final step: from the top representation it predicts the next token, the engine of text generation. Stack enough of these blocks, train on enough text, and that simple next-token objective produces the fluent, knowledgeable behaviour we call an LLM.
To see how this architecture turns into a working chatbot through training and generation, read how LLMs work, and for a deeper look at the tokenisation step, see what a token is.