OpenAI steered GPT-3 with human feedback so a tiny model beat a huge one

Making a language model larger made it more capable on benchmarks but not necessarily more helpful, truthful or safe. A model can generate confident nonsense or toxic text and still score well on what it was trained to do, so raw size was the wrong lever for following a user's intent.

OpenAI's answer was to go through humans instead of adding parameters. Labelers wrote demonstrations of the behavior they wanted, which were used to fine-tune GPT-3; then labelers ranked several model outputs, and those rankings became a reward the model was further tuned to maximize.

The resulting InstructGPT was far more aligned: in human evaluations its outputs were preferred to outputs of the 175B GPT-3, even though the winning model was 1.3B parameters, roughly a hundred times smaller.

It also got more truthful and less toxic while barely losing ground on public NLP benchmarks. The point was not the size of the model but the quality of the feedback collected for it.

Score outputs on human preference, not size

Human preference rankings give a clean reward for 'helpful, honest, harmless' that next-token prediction on raw text never captures.

Preference data is far cheaper than model parameters, so a relatively small amount can move a model a long way.

The same method works across tasks because it tunes the model's judgment, not a single skill.

Improving alignment made the better, smaller model the one people actually wanted to use, cutting serving cost at the same time.

Alignment is a data-and-feedback problem, not just a scale problem — spending your budget on collecting the right signal can beat spending it on more parameters.

Reinforcement learning from human feedback became the standard way to turn a raw pretrained model into an assistant, and it is the technique behind the chat assistants of the mid-2020s. The idea that collecting preference data can be worth more than scaling parameters reshaped how model providers spent their effort and compute.