The encyclopedia · R&D & Science · Technical decision · 2022
OpenAI steered GPT-3 with human feedback so a tiny model beat a huge one
InstructGPT fine-tuned GPT-3 on human demonstrations and rankings, so a 1.3B model was preferred to 175B GPT-3 on following instructions.
OpenAI
the move
Making a language model larger made it more capable on benchmarks but not necessarily more helpful, truthful or safe. A model can generate confident nonsense or toxic text and still score well on what it was trained to do, so raw size was the wrong lever for following a user's intent.
OpenAI's answer was to go through humans instead of adding parameters. Labelers wrote demonstrations of the behavior they wanted, which were used to fine-tune GPT-3; then labelers ranked several model outputs, and those rankings became a reward the model was further tuned to maximize.
The resulting InstructGPT was far more aligned: in human evaluations its outputs were preferred to outputs of the 175B GPT-3, even though the winning model was 1.3B parameters, roughly a hundred times smaller.
It also got more truthful and less toxic while barely losing ground on public NLP benchmarks. The point was not the size of the model but the quality of the feedback collected for it.
why it works
- Human preference rankings give a clean reward for 'helpful, honest, harmless' that next-token prediction on raw text never captures.
- Preference data is far cheaper than model parameters, so a relatively small amount can move a model a long way.
- The same method works across tasks because it tunes the model's judgment, not a single skill.
- Improving alignment made the better, smaller model the one people actually wanted to use, cutting serving cost at the same time.
what transfers
Alignment is a data-and-feedback problem, not just a scale problem — spending your budget on collecting the right signal can beat spending it on more parameters.
what came after
Reinforcement learning from human feedback became the standard way to turn a raw pretrained model into an assistant, and it is the technique behind the chat assistants of the mid-2020s. The idea that collecting preference data can be worth more than scaling parameters reshaped how model providers spent their effort and compute.
references
- Training language models to follow instructions with human feedback
- Reinforcement learning from human feedback
spotted an error? The archive wants to know.