D2PO: Discriminator-Guided DPO with Response Evaluation Models

Summary

This paper propose D2PO, discriminator-guided DPO, an approach for the online setting where preferences are being collected throughout learning. By collecting golden preference annotations during online sampling, D2PO simutenously trains the policy and the reward models.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The idea is quite potential, and the designed synthetic experiments are good. - The comparison of different discriminators is worth presenting.

Reasons to reject

- What is OPO and how is it implemented? OPO is not a well-established term in RLHF. I went through the two cited papers. Liu et al. use rejection sampling to collect preference pairs with the policy model, then apply DPO or SLiC loss for preference optimization. Lee et al. proposes a self-judgment approach where the reward model and policy model are initialized together. So it is not clear what OPO is referring to in this paper. From Figure 1, I guess OPO first trains an RM with offline preference data, then labels on-policy data with the static RM. However, it is necessary to clarify the loss function and specific implementation details. - The efficiency issue. For real tasks, the cost of acquiring on-policy gold labels would be high (e.g. human annotation). So the feasibility of this method is questionable. - The online RL methods such as PPO did not appear as a baseline, which also updated value models during training, but did not require gold labels. - There are three synthetic experiments with only one real-world experiment. It would be better to test the method on more real benchmarks. For example, you can try math or coding problems where the gold labels are easy to get. Overall, I think the main assumption of this work, better utilization of on-policy data, would make contribution to the field. I will be happy to increase my score once I see more convincing real-world experiments.

Summary

While direct preference optimization (DPO) gains popularity, it is often performed on static preference data collected ahead of the training, limiting its theoretical capability. This paper proposes D2PO, an online version of DPO that annotates preference data on-the-fly during training through an iteratively updated reward model. To achieve this, D2PO requires periodically collecting gold preference labels and retraining the reward model. Under a fixed budget, the authors show that D2PO achieves the best performance over several baselines on several diverse tasks.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

The proposal of D2PO that achieves promising alignment performance on several tasks, demonstrates the significance of online and on-policy learning for LLM alignment

Reasons to reject

- Clarity should be improved. - Claims are not always supported by the experimental results. - One advantage of DPO is to drop the separate reward model which greatly simplifies the training pipeline. D2PO, however, brings it back and relies on iteratively updated reward modeling, making the whole pipeline more complicated.

Questions to authors

* The “discriminator” in D2PO is actually the reward or preference model, but this term is very confusing in the context of LLM alignment. * In Section 6.1, the authors argue that the updated reward model improves the reward accuracy. However, the accuracy in Contrastive Distill doesn’t change that much comparing Figure 5 and 6. Could you please explain why D2PO still outperforms DPO w/ RM? * In Figure 3 and 4, why does OPO w/ gold reward model underperforms D2PO? If D2PO benefits from improved reward accuracy, gold rewards always give perfect reward for the training, but OPO w/ gold reward performs even worse. * With a fixed budget, how does the iteration T_p affect the performance? Should we perform more iterations, each annotating a smaller amount of gold preference data, or fewer iterations? * It’s not very clear how gold preference and silver preference data are used during the training, and why they have different amounts as shown in Table 1.

Summary

Current reinforcement learning approaches in alignment learning of LLMs are fundamentally constrained by the inability of reward models, which are generally trained with an initial set of human-labeled gold preferences, to generalize to the shifting distributions of policy models, often leading to susceptibility to reward hacking and bias towards superfluous features such as the response length. The paper offers a thorough investigation of the discriminator settings when the budget for online preference labeling is limited, and thus contributing to the field of online RL. Specifically, the paper proposes a training framework that alternates between training the discriminator/reward model using fresh data sampled from the latest policy model annotated with gold preferences and training the policy model with the dynamically updated reward model. The paper also proposes training the policy model with DPO in each alternating training step. The experiment results on synthetic problems such as writing-with-keywords and noun maximization and on real datasets (UltraChat) support the claim that the proposed method D2PO (essentially an online variant of DPO with a dynamically adjusted reward model trained with a fixed total budget of gold preference labeling) enables the policy model to reach the target reward more efficient and effectively.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The paper is well-motivated and provides sufficient background to justify its research statement and implications. - The experimental settings, including the synthetic and real-world datasets, are well-designed to support their claim. The results are also promising, providing more weight on online RL. - The method's key differentiations with the closely related work, "self-rewarding language models" (Yuan et al., 2024), include the incorporation of gold preference labeling throughout the online training and the discriminator setting (whether to be unified with the policy model, etc.). They provide a unique perspective on online RLHF, and the findings will be very useful for AL practitioners.

Reasons to reject

- The paper fails to explore when the budget on gold preference labeling is larger -- e.g., investigating whether having more initial gold preferences diminishes the benefit of subsequent iterative online training with silver preferences.

Questions to authors

What seed prompts did you use for generating responses to label with the reward model in each self-training step?

Reply

Thanks for your reply. I think the additional results and explanation are beneficial. I will maintain my score and lean positive to the paper.