What overfitting is
Overfitting happens when a model fits its training data too closely — capturing not just the genuine patterns but also the random noise and idiosyncrasies of that particular sample. The result is a model that scores impressively on the data it was trained on yet performs poorly on new, unseen data. In effect the model has memorised the training set instead of generalising from it, and generalisation to new data is the only thing that matters in deployment.
The opposite failure: underfitting
Overfitting has a mirror image. Underfitting is when a model is too simple, too constrained, or trained too briefly to capture the real structure in the data, so it does badly on both the training set and new data. The two failures bracket the target: an underfit model has not learned enough, an overfit model has learned too much of the wrong thing, and good performance lives in the sweet spot between them where the model captures the signal but not the noise. This balance is often described as the bias-variance trade-off: underfitting is high bias, overfitting is high variance.
How to detect it
The standard diagnostic is to watch training loss and validation loss together as training proceeds. Early on both fall — the model is learning real patterns. The warning sign of overfitting is when training loss keeps dropping while validation loss flattens and then starts to climb. The widening gap between the two curves is the model improving on data it has seen at the expense of data it has not. This is why you always hold out a validation set the model never trains on; without it, you cannot see overfitting happen.
How to prevent it
Several techniques, often combined, keep models from overfitting:
- More and more varied data. The single most reliable fix. With enough diverse examples, the noise averages out and the model has to learn the real signal.
- Weight decay (L2 regularisation). Adds a penalty for large weights to the loss, discouraging the model from relying too heavily on any single feature.
- Dropout. Randomly disables a fraction of units during each training step, forcing the network to spread its representation rather than memorise specific paths.
- Early stopping. Halts training at the point validation loss is lowest, before it begins to rise.
- Simpler models and data augmentation. Reducing capacity, or synthetically expanding the data with realistic variations, both reduce the room to memorise.
Why it matters for LLMs and fine-tuning
Overfitting is especially relevant when fine-tuning a model on a small dataset: with few examples and many parameters, a model can quickly memorise the fine-tuning set and lose its general ability. Watching the validation curve, using a low learning rate, and stopping early are the practical defences. Understanding overfitting is what separates “the model scored well in training” from “the model will work in production.”