Paper digest: Preference alignment — RLHF to DPO in plain English

Lineage focus: Christiano et al. — Deep Reinforcement Learning from Human Preferences · Ouyang et al. — Training language models to follow instructions with human feedback (InstructGPT) · Rafailov et al. — Direct Preference Optimization (DPO)
Sources: arXiv:1706.03741 · arXiv:2203.02155 · arXiv:2305.18290

One line claim

“Alignment” in modern LLM products usually means preference tuning: teach a model to prefer answers humans (or AI judges) rank higher, not only to predict the next web token. RLHF did that with a reward model plus reinforcement learning. DPO showed you can often skip the RL loop and fit the same preference story with a cleaner closed-form objective — if your preference data is real and your eval is not a vibe check.

RLHF and DPO do not make a model truthful by magic. They make it better at winning the preference label you actually collected.

Operator translation

Why this lineage exists

Pretrained language models learn distributional structure from internet-scale text. That is powerful and misaligned with what users want in a product: follow instructions, refuse harmful asks, prefer concise helpful answers, match a brand voice, avoid junk. Supervised fine-tuning (SFT) on demonstration data helps, but demonstrations are expensive and do not express trade-offs (“A is safer than B even if B is flashier”).

Preference-based methods start from comparisons. Given prompt x and two completions y⁺ and y⁻, humans (or rubrics) say which is better. That signal is richer per hour of labour than writing ideal answers from scratch, and it maps onto classical learning-from-preferences work in RL.

This digest is for operators and builders who hear “we use RLHF/DPO” in vendor calls. It stays with the canonical papers. For multimodal RL post-training rather than chat preference, see LaViDa-R1. For the Transformer base everyone fine-tunes, see Attention Is All You Need. Speech products that preference-tune on top of bad ASR inherit garbage — see Caribbean speech ASR.

Method, in plain English

Stage 0 — Pretrain + SFT

Start with a pretrained LM. Run supervised fine-tuning on high-quality instruction–response pairs so the model has a stable “helpful assistant” mode. InstructGPT (Ouyang et al., 2022) treats this as the necessary warm start before preference optimisation. Without SFT, RL or DPO often fights a base model that still wants to complete web text.

Stage 1 — Collect preferences

Show labellers a prompt and multiple model samples (or model vs demonstration). Record rankings or pairwise winners. Christiano et al. (2017) earlier showed deep RL from human preferences in control environments: humans compare trajectory segments; a reward model fits those comparisons; a policy maximises the learned reward. LLM alignment copies that template into language.

Stage 2a — RLHF classic loop (InstructGPT-style)

1. Fit a reward model rθ(x, y) that scores completions so preferred answers get higher reward. Training usually uses a Bradley–Terry style loss: probability that y⁺ beats y⁻ is a sigmoid of the reward difference.
2. Optimise the policy (the LLM) with reinforcement learning — typically PPO (Schulman et al.) — to maximise expected reward.
3. Keep a KL penalty toward a reference model (often the SFT checkpoint) so the policy does not drift into reward-hacking gibberish that fools the reward model.

That trilogy — SFT model, reward model, PPO policy — is what most people mean by RLHF for chat models. Anthropic’s Constitutional AI / HH-RLHF line (Bai et al.) extends the preference story with AI-assisted critique and harmlessness datasets; the engineering shape remains preference → reward → RL.

Stage 2b — DPO (skip the RL machinery)

Rafailov et al. (arXiv:2305.18290) ask: if the optimal RLHF policy under a KL-constrained reward objective has a closed form, can we solve for it directly from preference data?

Intuition. RLHF’s optimal policy can be written so that the reward is a function of the policy’s log-probabilities versus a reference model. Substitute that expression back into the Bradley–Terry preference model and the reward model disappears. You get a classification-style loss on pairs (y⁺, y⁻): raise likelihood of preferred answers and lower likelihood of dispreferred ones, with a temperature/β controlling how far you may move from the reference.

What operators feel. One training run on preference pairs. No sampling loop inside PPO. No separate reward network to host. Easier to debug when it fails — though not immune to bad data.

DPO is not the only “RL-free” preference method (IPO, KTO, ORPO, and cousins exist). It is the clearest teaching successor to InstructGPT-style RLHF in the open literature.

Results that moved

Christiano et al. (2017). Demonstrated that sparse human comparisons can train complex behaviours in RL domains where designing a reward by hand is hard. The paper is not an LLM paper; it is the conceptual parent of reward-from-preferences.

InstructGPT / Ouyang et al. (2022). Showed that a smaller GPT-3 variant fine-tuned with human feedback was preferred by labellers over much larger pretrained baselines on instruction-following prompts. Headline product lesson: alignment data can beat raw scale on the metrics users notice. They also document limits: preferences are subjective; models still hallucinate; improving helpfulness can trade off against truthfulness if labels reward confident answers.

DPO / Rafailov et al. (2023). Report matching or beating PPO-based RLHF on controlled preference and downstream instruction benchmarks with simpler training, across summary and dialogue-style tasks in their experimental setups. Ablations emphasise reference model choice and β. The community result was cultural as much as numeric: preference tuning became accessible to labs that could fine-tune but could not stably operate PPO.

What “better” meant in InstructGPT tables. Labellers preferred InstructGPT outputs on prompts probing instruction following, truthfulness-related questions, and toxicity-related behaviour relative to the base GPT-3 family in the paper’s studies — with the important caveat that different axes can move differently. Helpfulness gains do not automatically maximise factual accuracy. That multi-axis tension is why serious alignment cards report separate evals instead of a single “alignment score.”

Multimodal echo. LaViDa-R1 is not DPO-for-chat, but it rhymes: post-training with group-relative policy gradients, rewards for correctness/IoU/edit quality, and explicit tricks when all samples in a group fail. The lesson travels: preference or reward signal design dominates architecture branding. Video stacks (video diffusion / world models) that later add human preference finetunes inherit the same label politics — beauty versus physical consistency.

Limits (the papers already warned you)

Reward hacking and sycophancy. Policies learn to please the reward model or the labeller distribution. That can mean flattery, unsafe compliance dressed as helpfulness, or verbose answers that win length-biased labels.

Preference ≠ truth. Pairwise winners can favour fluent falsehoods. InstructGPT-style papers are explicit that human preference is not a factuality oracle. Grounding and retrieval are separate systems.

Annotator pool bias. Demographic and regional skew in labellers becomes policy skew. For Caribbean products, anglophone-US preference data will punish valid Creole-influenced phrasing unless you collect local comparisons on purpose.

KL / β under-tuned. Too little constraint → collapsed or hacky policies. Too much → expensive SFT with makeup. DPO’s β is not “set and forget” across domains.

Distribution shift. Preferences collected on general chat prompts transfer poorly to legal, medical, or finance desk work without domain pairs. The same holds for creative-ad assistants that must follow brand constraints.

Eval theatre. Win rates against a baseline with GPT-4-as-judge can be gamed by style. Keep a frozen human eval set and task metrics (exact procedure steps, refusal tests, citation correctness).

Online vs offline preference data. Classic RLHF often mixes offline human rankings with on-policy samples from the current model so the reward model sees the policy’s actual mistakes. Pure offline DPO on stale pairs can under-correct new failure modes after the policy drifts. Production recipes therefore refresh preference batches, not only train once and ship forever.

Safety is a preference mix, not a switch. Harmlessness and helpfulness compete in pairwise labels. Bai et al.’s HH-style datasets and Constitutional approaches try to structure that trade-off; they do not remove it. Operators who only reward “always answer” will get compliance. Operators who only reward refusal will get a brick wall. Write the mix down.

Who should care

Founders buying “aligned” APIs. Ask for the recipe: SFT only, RLHF (PPO), DPO, or proprietary variant? Ask what preference data covers and whether refusal / safety was joint or bolted on. Ask for win-rate and hallucination / harm evals.

ML engineers. If you can label a few thousand clean pairs in-domain, DPO (or cousin) is often the first preference lever to try after SFT. Invest in data quality before building a PPO platform. Keep a reference checkpoint and track KL-like drift even when the loss is DPO.

Policy and safety reviewers. Alignment here is preference optimisation, not proof of value alignment with society. Documents should say which humans’ preferences were optimised.

Caribbean operators and small studios. Your CS voice, pricing honesty, and WhatsApp tone are preference problems. You can often get farther with a small local preference set on top of a strong base model than with a giant general chat model that “sounds American polite.” Pair this digest with speech ASR so you do not preference-tune on broken transcripts, and with video diffusion / world models when the product is generative video rather than chat — different stacks, same lesson: specify the objective you actually reward.

Bottom line

RLHF made human pairwise preferences a first-class training signal for instruction-following LMs via reward models and PPO. DPO showed the same preference model can often be optimised directly, cutting moving parts. Both succeed or fail with the data: whose taste, which prompts, what trade-offs. For operators, “aligned” should unpack to a named method, a named preference source, and an eval that can lose.

We report papers. Digests written at Yatishara Blog; not a product review.