Preference Learning Algorithms Do Not Learn Preference Rankings

Summary

This paper investigates the effectiveness of preference learning algorithms, include the RLHF (PPO-based) and DPO in training LLMs. In particular, the authors study the non-regularized ranking accuracy, which is largely ignored in the previous RLHF literature. The authors approach this investigation through both theoretical analysis and empirical experiments and show that whether the ranking is correct or not highly depends on the initial reference policy.

Strengths

The research problem is interesting. While the RLHF and DPO mainly care about the KL-regularized target, it is debatable that the likelihood of the response under the resulting policy is what we are really interested in. Therefore, the notion of ranking policy in terms of this non-regularized target deserves more attention. With the goal in mind, the DPO loss target in definition 2.2 clearly shows that the log-ratio on the reference model serves as an adaptive margin for the data point. Therefore, the discussions around the reference model is natural and informative. I particularly appreciate the section 4 on the empirical studies of DPO, which can largely help us to understand the dynamic of this widely adopted algorithm. Overall, the authors identify an interesting question and the present a complete story with solid evidence in both theory and experiments.

Weaknesses

My major concern is that this paper only studies the goal of helpfulness in chat style. First, while it is fair to say that DPO is not good at optimizing the ranking policy in terms of the non-regularized target, it is not immediately clear that the non-regularized ranking policy is a better metric. For instance, a concurrent work (so this should not hurt the novelty of this paper) [1] proposes simpo with many shared insights. In particular, they also notice the mismatch between the ranking policy and the target of DPO. However, in their experiments, as the win rates evaluated on the standard benchmark significantly increase, we also observe a significant drop in some academic benchmarks, particularly those on the mathematical problems solving. Based on my personal experience, if we adopt a smaller KL coefficient (e.g., 0.01, so less regularization), the final model performance is worse than the model with a moderate one (say 0.1). This is particularly true for the online iterative learning, which we are more interested in as the authors also notice that these variants outperform the offline one with a large margin. In most of the cases, the ability of maximizing the reward/ranking accuracy, while maintaining a moderate KL divergence is more suitable to evaluate the algorithm performance in my viewpoint. Therefore, the ranking accuracy alone is also not a good learning target, which is also evidenced by figure 4 of this work. Based on the above two points, I feel that the paper can still be largely improved if a more comprehensive evaluations can be done, like the trade-off between the performance on instruction-following benchmarks (alpaca-eval, mt-bench, gpt4 win rate) and other academic benchmarks. But overall, I support the acceptance. [1] SimPO: Simple Preference Optimization with a Reference-Free Reward

Questions

see weakness

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

yes

Summary

The paper investigates the effectiveness of preference learning algorithms, RLHF and DPO, in training LLMs to rank human-preferred outputs higher. It challenges the assumption that these algorithms can successfully teach models to rank outputs according to human preferences. Empirical findings show that most state-of-the-art models achieve low ranking accuracy, often below 60% on standard datasets. The paper identifies a significant alignment gap due to the limitations of the DPO objective and provides a theoretical analysis of the difficulty in correcting ranking errors. It concludes by highlighting the need for a reevaluation of these algorithms and calls for more nuanced analyses of preference training dynamics.

Strengths

1. The paper provides a thorough empirical analysis, revealing the limited effectiveness of RLHF and DPO in improving ranking accuracy, which is a significant contribution to the field. 2. It offers theoretical insights into why these algorithms fail to correct ranking errors, providing a deeper understanding of the underlying issues in preference learning for LLMs. 3. The findings have clear practical implications for the development of alignment techniques in LLMs, motivating the need for more effective methods to train models that better reflect human preferences.

Weaknesses

1. While the paper suggests a link between preference learning and win rate, it would be strengthened by a more explicit exploration of how these metrics interrelate. Specifically, the authors could benefit from a deeper analysis of the implications of using ranking accuracy versus win rate as measures of model performance, including potential discrepancies and how they might be reconciled. 2. The paper identifies a significant determinism of ranking accuracy by reference model likelihoods, which is a crucial observation. However, it would be advantageous for the authors to expand on this point by discussing potential techniques or algorithmic adjustments that could alleviate the heavy reliance on the reference model. This would bolster the paper's contribution by addressing a key challenge in the field of preference learning for LLMs.

Questions

See Weaknesses.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes, the authors fully address the limitations and potential social implications of their work.

Summary

This paper studies the preference learning algorithms like DPO in Large Language Models (LLMs). It focuses on the ranking accuracy and shows that existing algorithms fail to achieve high ranking accuracy. Then, the paper further proposes an idealized ranking accuracy and finds that there is an alignment gap.

Strengths

- A metric called ranking accuracy and a variant of this metric are proposed. - Extensive empirical observations of DPO are provided, which may be interesting for algorithm analysis and understanding.

Weaknesses

- The motivation of studying the rank accuracy is weak and not justified well. - The analysis and results are somewhat superficial.

Questions

The contributions, compared with similar works, are not clear for reviewers to give an acceptance recommendation. [1] Xu, Shusheng, et al. "Is DPO superior to PPO for LLM alignment? A comprehensive study." arXiv preprint arXiv:2404.10719 (2024). [2] Tajwar, Fahim, et al. "Preference fine-tuning of LLMs should leverage suboptimal, on-policy data." arXiv preprint arXiv:2404.14367 (2024). [3] Tang, Yunhao, et al. "Understanding the performance gap between online and offline alignment algorithms." arXiv preprint arXiv:2405.08448 (2024). The analysis of PPO in terms of ranking accuracy is missing, which somehow makes the conclusion regarding on-policy and off-policy weak.

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

- The motivation for studying ranking accuracy is weak. The ranking accuracy, especially when defined on the offline dataset, usually does not correlate with the quality of generation behaviors. Let us split the response set into two parts, R_good and R_bad. In practice, we care that sum\_{y} P( y in R_good) > sum\_{y'} P(y' in R_bad). The proposed ranking accuracy, instead cares about P(y in R_good) > P(y' in R_bad), where y and y' are two events from R_good and R_bad, respectively. Thus, a good generative model can provide many good responses with high probability and, at the same time, it may have a low ranking accuracy. This says that ranking accuracy measures the worst-case behavior of generative models. Thus, we do not need to bother ourselves to care about the ranking accuracy, especially for a small dataset where only a few pairs of preference are given. - The analysis in terms of ranking accuracy is somehow superficial. In fact, it is the algorithmic bias that DPO introduces the KL-regularized optimization, and as such, it cannot recover the preference-generating distribution. Briefly, the BTL assumption does not invovle the KL term. See the recent work in [1] for more discussion (since this work was posted after NeurIPS submission deadline, it does not affect the review decision). This can also be reflected in equation (2): assume that pi_REF is a uniform policy so its effect can be eliminated and the parameter beta = 1, we see that it recovers the preference-generating policy in terms of alpha. The BTL model assumes that alpha + 1 - alpha = 1, however it may not be true that pi*(y\_w|x) + pi*(y\_l|x) = 1. The procedure I mentioned is also reflected in equation (4), where the authors introduce a new loss. It in fact corresponds to add a entropy regularization to eliminate the bias of pi\_ref. Based on the considerations mentioned above, the reviewer does not clearly perceive the technical depth and the contributions of this paper. [1] Xiao, Jiancong, et al. "On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization." *arXiv preprint arXiv:2405.16455* (2024).

Summary

The paper empirically highlights a few potential flaws in RLHF and DPO that prevent preference-tuned models from achieving high ranking accuracy. It presents a collection of empirical and theoretical findings: + Existing preference-tuned models achieve low ranking accuracies. + The idealized policy and the DPO/RLHF policy has a significant gap in their ranking accuracy. + Preference learning algorithms such as DPO rarely flip incorrect rankings throughout the training process. + Ranking accuracy and win rate are positively correlated when models are close to the reference model.

Strengths

The paper provides a thorough empirical and theoretical examination of the performance of preference learning algorithms, particularly RLHF and DPO, in aligning LLMs with human preferences. The paper highlights the significant alignment gap between the observed and idealized ranking accuracies. The empirical study is extensive, covering a variety of state-of-the-art preference-tuned models across multiple datasets, providing robust evidence supporting its claims on the gap between the idealized and learnt policy under the DPO/RLHF objective.

Weaknesses

1. While it is already argued in the paper that ranking accuracy is practically more meaningful than the reward accuracy, it is still questionable whether totally discarding $\pi_{ref}$ is reasonable. Specifically, the paper should have included experiment results on whether there is also a gap in reward accuracy between the learnt and ideal policy. After all, the DPO and RLHF algorithm explicitely maintain the reference model $\pi_{ref}$, and the implicit reward should be indicative on human preference. 2. Due to the small size of preference labels for any given response pair (1 for most dataset, and 4 for the Alpaca Farm as stated in Sec C.4), it is more reasonable to consider the soft ranking accuracy instead of the hard ranking accuracy. The authors could have reported both or discuss on whether hard ranking accuracy is more meaningful.

Questions

1. One concern is that the ranking accuracy considered here is the hard accuracy (accuracy of the Bayesian optimal classifier). I wonder if the "soft" ranking accuracy will exhibit more noticable improvement after preference fine-tuning. The soft ranking accuracy is $\frac{\pi(y_w)}{\pi(y_w) + \pi(y_l)}$ instead of hard thresholding the predicted accuracy. 2. It is not clear to me what is the main failing part in DPO/RLHF training for high ranking accuracy. In Theorem 3.1, it seems that the perfect RLHF policy should encode the information from $\pi_{ref}$ and the dataset perfectly at the same time. Does the DPO/RLHF model fail to capture $\pi_{ref}$ or the preference dataset? Have the authors tried to plot the quantity $\frac{\pi_{DPO}(y_w)\pi_{ref}(y_l)}{\pi_{DPO}(y_w)\pi_{ref}(y_l)}$ and compare it to $(\frac{\alpha}{1-\alpha})^{\beta}$ (perhaps a scatter plot)? This can illustrate more clearly what DPO/RLHF failed in reaching the perfect policy.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A