Summary
RLHF is generally done by training a reward model on a dataset of preferences and then using on-policy RL to finetune a language model with the reward model. This work proposes to bypass explicit learning of a reward model and finetune the language model directly on the preference dataset. This is conceptually similar to offline RL but the authors derive a direct preference objective such that the training is supervised not RL-based.
Using a standard preference model, a supervised objective (DPO) is derived, theoretically justified, and compared to PPO. The authors test their method on three tasks: sentiment completion (IMDB), summarization (as in Stiennon et al), and single-turn dialogue with Anthropic's HH dataset. They find that DPO achieves higher reward while staying closer to the reference model (KL) in the IMDB task. Using GPT-4 as a proxy for human judgement on summarization, DPO can match PPO's performance while being less sensitive to sampling temperature. They validate GPT-4 with human annotators and find that DPO is even slightly more preferred over PPO. On dialogue, they find that DPO outperforms supervised and re-ranking (again using GPT-4 eval) but do not compare against PPO.
Strengths
The overall idea is relatively simple but the justification and theorems provide an excellent basis. The paper is timely and gives an excellent alternative to RLHF, that should be notably more efficient. The paper is clearly written and has the potential to be very impactful if used at scale in modern methods. GPT-4 evaluations and the follow-up human evaluations are strong and demonstrate both improved performance and robustness which can be notably bad in RLHF. The authors include the minimal code for DPO in the appendix which is commendable and many GPT-4 evaluation details for excellent reproducibility. The presentation is also very clear and breaking down the DPO loss into its components with clear language is an excellent addition.
Weaknesses
The main weakness is the unexplored main difference between DPO and PPO: generalization. The authors note it in limitations but since PPO is trained on-policy and DPO is trained offline, looking at generalization outside the dataset would be helpful. The work, as is, feels sufficient without those experiments but the difference does play into an inconsistency with some theorems. The other major issue are some incongruenties in the evaluation that the authors should respond to.
The proofs and theorems contain a hidden assumption that is not explicitly stated. In the formulations for PPO, the optimization uses $y \sim \pi_\theta$ whereas in the formulations for DPO $y \sim D$ i.e. DPO's algorithm is offline and the model learned from its implicit reward has guarantees over the dataset. In contrast, PPO's algorithm is learned over a new set of datapoints sampled from the model during training. This means that Equation 3 is maximized over a different set of data points than Equation 7. So even though Theoerem 1 is correct, Lemma 2 does not fully encompass this situation. With offline DPO vs on-policy PPO, equivalent reward models could induce different optimal policies since they are being optimized over a different set of points. The section seems to imply that PPO and DPO can induce the same optimal policies because of the ambiguity with $y$. Clarifying this would be helpful. This also means that section 5.2 and the substitution used to make Equation 10 is not exactly accurate as the optimal policy for PPO is not the same as the optimal policy for DPO. I still feel the point of section 5.2 is reasonable but the authors should explicitly note the issue with the substitution.
There are issues with each of the different evaluations, even though the overall picture does suggest that the method is sound. I will increase my score if the authors simply clarify some questions about the evaluations (not even necessary to do changes / re-run experiments)
For controlled sentiment (IMDB), the authors do not give many details, seem to deviate from previous work on the task without explanation. The GRUE benchmark (RL4LMs) specifically uses GPT-2 base and finds that models should not be supervised-finetuned before RL training. In contrast, the authors use GPT-2 large (not even GPT-2 medium) and do supervised finetuning before PPO. No details of the baseline are given except for the graph legend which implies the PPO baseline comes from the `trl` library, although the authors only claim to use the `trlx` library. It is also unclear whether the evaluation is over the train set or the test set. Since this task is the only one where the authors show the tradeoff between reward achieved and KL, these details seem important.
For summarization, the issue is relatively minor. GPT-4 may be an unfair evaluator (see Large Language Models are not Fair Evaluators) and the work would benefit from evaluating samples in both positions (i.e. option A then B and option B then A)
For Anthropic's HH, the authors claim there is no SFT model available but there are many available pretrained models for this benchmark that the authors do not compare against. The most prominent example being a GPT-J 6B model `reciprocate/ppo_hh_gpt-j` hosted on huggingface which also has wandb runs and repro code available on the `trlx` library. The authors have chosen Pythia 2.8B when there exists a similar Pythia 1.6B PPO-trained on the HH dataset and it is unclear why the authors do not compare to this model or another PPO-trained model.
Questions
What are the compute differences between DPO and PPO? It feels like DPO should be much more efficient and this information would only strengthen your work.
Why do you not compare against ILQL for sentiment? As an offline RL method, it seems the closest RL comparison to DPO
For reproducibility, what exact API are you using for the GPT-4 evaluation (e.g. GPT-0314)
Can you move Figure 2 and 3 to the next pages so it is closer to the text that references it?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors did quite well to address limitations in the paper, I believe they covered most of my concerns there.