Summary
The authors propose Cal-DPO, a preference-tuning algorithm that modifies the DPO loss by adding two MSE terms that aim to "calibrate" the log-likelihood ratios of y_w and y_l to their respective reward values. They claim that this is advantageous because the DPO loss only maximizes the reward ratio, and does not constrain the respective reward values. They also claim that Cal-DPO exhibits a "negative gradient" (it pushes down the likelihood of undesirable outputs) and mode-seeking behavior. They evaluate their methods by training the Zephyr 7B model on Cal-DPO and DPO, and comparing the results on a few different benchmarks. Cal-DPO generally seems to outperform DPO and other related variants.
Strengths
- Cal-DPO has some solid motivations -- it is true that DPO does not constrain the reward margin or the absolute scale of the rewards, which often leads to issues with over-optimization. IPO was similarly motivated in that it explicitly constrains the reward margin.
- The concept of the "negative gradient" is a well-established motivation for designing training objectives that decrease the likelihood of y_l.
- Cal-DPO is easy to implement and can be straightforwardly combined with other training objectives.
Weaknesses
- **Unfair comparisons to other methods**: The authors show in Figure 4 that the best $\beta$ value for Cal-DPO is 0.001, and they choose to use this same value for their comparisons to all other methods (described in Appendix B.1, where they state that "all the hyperparameters are set to be the same"). However, this implies that hyperparameter tuning was conducted for Cal-DPO, but not for any of the other methods. In fact, past work indicates that the optimal $\beta$ value can vary widely depending on the combination of model and algorithm, in some cases reaching as high as 0.6 (see https://huggingface.co/blog/pref-tuning). The difference in accuracy/performance can be vast, depending on the hyperparameter configuration. In general, the fairer way to compare methods is to conduct a hyperparameter search separately for every method, rather than to fix a set of hyperparameters. This is especially important in this case, since the $\beta$ was already selected to be the optimal one for Cal-DPO. Without this procedure, it is unclear whether Cal-DPO truly outperforms the other methods, or if the other methods were sub-optimally trained.
- The authors train the model on a set of training datasets covering safety behaviors, summarization, and sentiment detection, but the evaluation benchmarks in Table 2 primarily cover other types of capabilities, such as math and abstract reasoning. These are not preference learning tasks, as they have exact answers, rather than subjective ratings of human preference. Even if a couple other papers have done this evaluation, this does not make as much sense as evaluating on more related benchmarks, such as MT-Bench. The Cal-DPO accuracy numbers in this table are also quite close to the other methods' -- and given the hyperparameter tuning issues, it is unclear how much of this difference is noise or is meaningful.
- The authors predicate much of their rationale on the claim that other techniques like DPO reduce the likelihood of the chosen response. However, since DPO also reduces the likelihood of the rejected response, it is unclear whether this effect is actually undesirable -- to answer this question, one would have to know where the probability mass gets moved to instead. Since we know that DPO does in fact improve win rate, it seems likely that there is probability mass moved to more preferred outputs. The authors also do not cite any evidence that this is necessarily undesirable.
- Additionally, not all y_w's in preference datasets are high-quality -- it is often the case that both y_w and y_l are low-quality, but y_w is slightly less low quality than y_l. In this case, it is not necessarily desirable for the model to increase the probability mass on y_w.
- Similarly, the authors state that "an undesirable consequence of this behavior is that the learned policy increases the likelihood of unknown out-of-distribution responses," but this is also unfounded. If there is past work or experimental evidence that supports this claim, it would be helpful to cite it here.
- The authors also claim that mode-seeking behavior is more desirable here -- however, one can also see cases where this is undesirable. Since mode-seeking encourages more probability mass to be placed on high-reward outputs, and DPO/Cal-DPO rewards are based off the current model's likelihood, this promotes a positive feedback cycle where the algorithm continuously places more mass on outputs that are already high-likelihood under the current model. This may cause fast reward over-optimization, which has been a frequently observed issue in RLHF-based algorithms.
- The experiments are only conducted on the Zephyr 7B model, and it would be much more convincing if the results reproduced to a couple other sizes and types of LLMs as well.
- The paper also does not seem to list the generation parameters used for any of the evaluation experiments.
Minor nits:
- The stated IPO objective in Eq. 18 is incorrect -- the constant term should be $\frac{1}{2\tau}$, not $\frac{1}{2}$. Also, there is no $\beta$ coefficient. Alternatively, one could rename $\tau$ as $\beta$ instead and write $L_{IPO}=(h_{\theta}(x,y_w,y_l)-\frac{1}{2\beta})^2$.
- What are the "Value"s on the y-axis in the left two plots of Figure 3?
Limitations
The stated limitations about Cal-DPO being limited to offline methods is reasonable. An additional limitation is that the experiments have only been conducted on the Zephyr 7B model, and evaluated for preference learning on only a couple of small datasets (e.g. TL;DR summarization and Anthropic HH).