The Direct Preference Optimization Family

Learning Objectives

• Derive the DPO closed form from the RLHF-with-KL optimum.
• State the failure mode each of IPO, KTO, SimPO, ORPO, BPO fixes in DPO.
• Distinguish "implicit reward gap" from "preference strength" and explain why IPO's identity mapping matters.
• Explain why Rafailov et al. (NeurIPS 2024) prove DAAs over-optimize despite having no explicit RM.

The Problem

The RLHF objective (Lesson 1):

\max_{\pi} \; \mathbb{E}_{x, y \sim \pi} \big[ r(x, y) \big] - \beta \, \mathrm{KL}(\pi \,\|\, \pi_{\text{ref}})

has a known optimum:

\pi^*(y \mid x) = \frac{1}{Z(x)} \, \pi_{\text{ref}}(y \mid x) \exp\!\left(\frac{r(x, y)}{\beta}\right)

So the reward is implicitly defined by the ratio of the optimal policy to the reference:

r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)

Substitute this into the Bradley-Terry preference likelihood and the partition function Z(x) cancels because it depends only on x. What remains is a loss in the policy parameters alone — no reward model needed. That is DPO.

The wrinkle: the derivation assumes the optimum is reachable, the preference data is in-distribution, and the reference policy is the true mode anchor. None of these hold exactly. Every family member fixes a different violated assumption.

The Concept

DPO (Rafailov et al., 2023)

L_{\text{DPO}} = -\log \sigma\!\left(
  \beta \log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)}
  - \beta \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}
\right)

What can go wrong:

• The implicit reward gap beta * (log(pi/pi_ref)_w - log(pi/pi_ref)_l) is unbounded. A tiny preference can produce an arbitrarily large gap.
• The loss drives chosen and rejected log-probs in opposite directions. It can push the chosen absolute log-prob down as long as the rejected falls faster. This is the Degraded Chosen Response phenomenon.
• Out-of-distribution preferences (pairs where both responses are rare under the reference policy) produce arbitrary implicit rewards.

IPO (Azar et al., 2024)

Identity Preference Optimization replaces the log-sigmoid with an identity mapping on the preference probability. The loss becomes a squared-error on a bounded target:

L_{\text{IPO}} = \left( \log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} - \frac{1}{2\beta} \right)^2

The margin is bounded by 1/(2 beta). Preference strength and implicit-reward gap are proportional. No blow-up.

KTO (Ethayarajh et al., 2024)

Kahneman-Tversky Optimization drops pairwise structure entirely. Given a single labeled output and a binary "desirable" or "undesirable" signal, it maps to a prospect-theory utility:

v(x, y) = \sigma\!\left(\beta \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)} - z_{\text{ref}}\right)

with different weights for gains and losses (loss aversion). Benefit: you can use unpaired data, which is far more plentiful.

SimPO (Meng et al., 2024)

Simple Preference Optimization aligns the training signal with generation. Remove the reference policy entirely and normalize log-likelihood by length:

L_{\text{SimPO}} = -\log \sigma\!\left(
  \frac{\beta}{|y_w|} \log \pi(y_w \mid x)
  - \frac{\beta}{|y_l|} \log \pi(y_l \mid x)
  - \gamma
\right)

with a margin gamma to stabilize. The length normalization removes the incentive to exploit DPO's length-bias failure mode (longer y_w gives a larger log-prob gap by construction).

ORPO (Hong et al., 2024)

Odds-Ratio Preference Optimization adds a preference term to the standard SFT negative log-likelihood:

\begin{aligned}
L_{\text{ORPO}} &= L_{\text{NLL}}(y_w) + \lambda L_{\text{OR}} \\
L_{\text{OR}} &= -\log \sigma\!\left(\log \frac{\text{odds}(y_w)}{\text{odds}(y_l)}\right)
\end{aligned}

No reference policy — the SFT term is the regularizer. Train in a single stage from the base model to the aligned model. No separate SFT checkpoint.

BPO (ICLR 2026 submission, OpenReview id=b97EwMUWu7)

Identifies the Degraded Chosen Responses problem: DPO preserves the ranking y_w > y_l but the absolute log-prob of y_w can drop. BPO adds a single-line correction that penalizes downward moves on the chosen response. Reported +10.1% accuracy on Llama-3.1-8B-Instruct on math reasoning over DPO.

The universal result: DAAs still over-optimize

Rafailov et al. "Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms" (NeurIPS 2024) trained policies with DPO, IPO, SLiC on multiple datasets across KL budgets. The gold-reward-vs-KL curves have the same Gao et al. peak-and-collapse shape. The implicit reward queries out-of-distribution samples during training; KL regularization does not stabilize this.

DAAs do not escape Goodhart. They change the surface where it bites from "reward model over-optimized" to "reference policy ratio over-optimized." The universal fix — better data, ensembles, early stopping — applies to both.

Choosing among them (2026)

• If you have large paired preference data: DPO with conservative beta, SimPO if length bias is evident.
• If you have unpaired binary feedback: KTO.
• If you want a single-stage pipeline from a base model: ORPO.
• If you see degraded chosen log-probs in DPO logs: BPO.
• If preference strengths vary widely and DPO is saturating: IPO.

Every lab runs all five on a battery and picks the winner per task. There is no reason the optimum is the same for math reasoning and safety.

dpo-margin

Use It

code/main.py compares six losses (DPO, IPO, KTO, SimPO, ORPO, BPO) on a toy preference dataset where the true preference strength varies by pair. Each loss is optimized against the same 500-pair sample with a small softmax policy. Plots final win rate, chosen-log-prob drift, and implicit-reward spread per method.

Ship It

This lesson produces outputs/skill-preference-loss-selector.md. Given dataset statistics (paired vs unpaired, variable vs uniform preference strength, length distribution) and a target (single-stage or SFT-then-preference), recommend a preference loss and report the failure mode it protects against.

Exercises

1. Run code/main.py. Report the final chosen-log-prob drop for DPO and BPO. BPO should retain higher chosen absolute probability — verify this.

2. Modify the preference data so that all pairs have equal strength. Which of the six methods is most robust? Which degrades? Explain IPO's advantage here.

3. Make the rejected responses on average 2x longer than chosen. Without changing anything else, show DPO's length exploitation numerically and SimPO's fix.

4. Rafailov et al. (NeurIPS 2024) claim DAAs over-optimize. Reproduce a single-point version: plot chosen-minus-rejected KL divergence and observe over-optimization in DPO at large beta.

5. Read the BPO paper abstract (OpenReview b97EwMUWu7). Write down the one-line correction BPO adds to DPO. Confirm against the implementation in code/main.py.

Key Terms

Further Reading

• Rafailov et al. — Direct Preference Optimization (NeurIPS 2023, arXiv:2305.18290)
• Azar et al. — A General Theoretical Paradigm to Understand Learning from Human Preferences (AISTATS 2024, arXiv:2310.12036) — IPO
• Ethayarajh et al. — KTO: Model Alignment as Prospect Theoretic Optimization (arXiv:2402.01306)
• Meng, Xia, Chen — SimPO (NeurIPS 2024, arXiv:2405.14734)
• Hong, Lee, Thorne — ORPO (EMNLP 2024, arXiv:2403.07691)
• BPO — Behavior Preservation Optimization (ICLR 2026 OpenReview b97EwMUWu7)
• Rafailov et al. — Scaling Laws for RM Overoptimization in DAAs (NeurIPS 2024, arXiv:2406.02900)