Reinforcement Learning from Human Feedback (RLHF) is the technique that turned raw language models into the helpful assistants people actually use. A model trained only to predict the next word will produce fluent text, but it will not reliably follow instructions, refuse harmful requests, or pick the most useful of many plausible answers. RLHF is how that gap is closed.
The starting point: a pretrained model
RLHF does not start from scratch. It begins with a large model that has already been pretrained on huge amounts of text and usually lightly fine-tuned on example instructions (supervised fine-tuning). This model is competent but unaligned — it knows a great deal and writes fluently, but it has no strong preference for being helpful or safe. RLHF then steers that competence toward what humans want.
Step one: collect human preferences
The first RLHF step is gathering comparison data. The model generates several responses to the same prompt, and human labellers rank them from best to worst. Crucially, labellers are not writing answers from nothing — they are choosing between options, which is faster, more consistent, and easier to do well. The output is a large dataset of “response A is better than response B” judgements across many prompts.
Step two: train a reward model
It is impossible for humans to score the millions of outputs that reinforcement learning needs. So those preference rankings are used to train a separate model called the reward model, whose only job is to look at a prompt and a response and predict the score a human would give it. Once trained, the reward model acts as a fast, automatic stand-in for human judgement.
Step three: optimise with reinforcement learning
Now the main model — the policy — generates responses, the reward model scores them, and a reinforcement learning algorithm such as PPO (Proximal Policy Optimization) nudges the policy’s weights to earn higher reward. A penalty term keeps the policy from straying too far from the original model, which stops it from discovering weird, incoherent outputs that happen to fool the reward model. Over many iterations, the model learns to produce answers humans consistently prefer.
Limits and what comes next
RLHF is powerful but imperfect. The model can learn to please labellers rather than be truly correct — a problem called reward hacking — and it can absorb the biases of whoever wrote the preferences. Newer methods like Direct Preference Optimization (DPO) skip the separate reward model and RL loop, training the model directly on preference pairs, which is simpler and more stable. Anthropic’s Constitutional AI adds AI-generated feedback against a written set of principles to reduce reliance on human labels. The shared insight across all of them remains the same: the fastest route to a model that behaves well is to teach it from human preferences, not just from raw text.