We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. We provide a theoretical explanation for this regularization effect using ideal DPO policy theory. We further show performance improvements over DPO in translation and mathematical reasoning using our DPO variants. We find it can be beneficial to include ties in preference optimization rather than simply discard them, as is done in common practice.
Paper
Similar papers
Peer review
Summary
This paper extends the Direct Preference Optimization (DPO) framework to handle tied preferences in pairwise comparisons by incorporating two established preference models, the Rao-Kupper and Davidson models. Unlike the original DPO, which discards ambiguous data points, the proposed variants, DPO-RK and DPO-D, allocate probabilities to ties, preserving more information from the data. The paper evaluates these DPO variants in machine translation and summarization tasks, showing that the inclusion of ties enhances regularization effects without compromising performance.
Strengths
1. The paper successfully integrates the Rao-Kupper and Davidson models to extend DPO for handling ties, addressing a known limitation in the original DPO framework where tied pairs were previously discarded. This extension makes the preference modeling process more inclusive and data-efficient. 2. The experimental analysis includes various statistical measures and visualization of training dynamics, such as reward margins and gradient scale factors, which add depth to the assessment of how well the models learn from tied data. 3. The paper is well-written and easy to follow. The adaptation of Bradley-Terry models for ties is systematically outlined, with clear mathematical formulations and derivations for the proposed DPO-RK and DPO-D models.
Weaknesses
1. The introduced method shows limited novelty, as it builds upon existing Bradley-Terry extensions to accommodate ties. The proposed DPO variants primarily involve substituting the standard Bradley-Terry model with these pre-existing extensions (Rao-Kupper and Davidson) for preference modeling, rather than presenting fundamentally new methodologies. 2. Although the proposed DPO variants improve regularization effects, they fail to demonstrate any clear enhancement in task performance relative to the original DPO. This raises questions about the practical value of the proposed modifications, particularly in performance-critical applications. 3. The experiments are limited to machine translation and summarization tasks, leaving more complex domains such as open-ended generation tasks (e.g., AlpacaEval, Arena-Hard, MT-Bench) and mathematical reasoning unexplored. Expanding evaluations to these challenging areas would provide a more comprehensive assessment of the proposed methods. 4. The paper introduces additional parameters, such as the probability adjustment factor $\upsilon$ for the Rao-Kupper and Davidson models, but lacks extensive discussion on the impact for tuning these parameters.
Questions
See weeknesses.
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Thank you for your review and suggestions. In response to your concerns: > The introduced method shows limited novelty, as it builds upon existing Bradley-Terry extensions to accommodate ties. The proposed DPO variants primarily involve substituting the standard Bradley-Terry model with these pre-existing extensions (Rao-Kupper and Davidson) for preference modeling, rather than presenting fundamentally new methodologies. We acknowledge that we are making use of well-established Bradley-Terry extensions. Our novelty lies in addressing a gap in the literature in that these extensions have been overlooked to date in the development of powerful algorithms like DPO and its associated variants. We also make contributions in the understanding of the effect of including ties by a theoretical analysis based on the ideal DPO simulation equation (Sec.3.1.1). > Although the proposed DPO variants improve regularization effects, they fail to demonstrate any clear enhancement in task performance relative to the original DPO. Thanks for raising this. We emphasize that the goal of the paper is not to improve over DPO(CP) but to understand the effect of adding ties to DPO and to investigate DPO variants based on these tie-compatible Bradley-Terry variants. We have identified a weakness inherent in DPO and related procedures that the Bradley-Terry assumption does not accommodate ties. Our proposed DPO variants address this by considering tie-compatible BT extensions. Specifically, when TPs are included in the training dataset, we see degradation in DPO. By contrast, we find improved regularization in terms of KL and task performance using our proposed variants > The experiments are limited to machine translation and summarization tasks, leaving more complex domains such as open-ended generation tasks (e.g., AlpacaEval, Arena-Hard, MT-Bench) and mathematical reasoning unexplored. Thanks for raising this. We find that even on simpler tasks like translation and summarization, the issues of ties are inherent to the DPO methodology. We acknowledge that this problem may also exist in more complex tasks, but given that the ties are more difficult to define and identify in mathematical reasoning datasets, we consider such datasets beyond the scope of our study and leave them as future work. > The paper introduces additional parameters, such as the probability adjustment factor υ for the Rao-Kupper and Davidson models, but lacks extensive discussion on the impact for tuning these parameters Thanks for raising this point. In our early experiments with a weaker translation model, we find that the choice of $\alpha$ does not have a strong impact on final task performance. To fully address your concern, we conducted additional experiments by varying the hyper-paramter $\nu=\exp(\alpha)$ and report the KL divergence, KIWI-22, BLEURT, and COMET below. We find that (1) the final task performance is stable over a range of alpha values (note that $\alpha<0$ is illegal for DPO-RK); (2) for large value of alpha, there is relatively little regularization as measured by KL divergence. Thank you for your suggestion. We will include this table in our revision. | System | alpha | KL | KIWI-22 | BLEURT | COMET | |--------------------|---------------|-------|---------|--------|-------| | **DPO-RK (beta=0.4)** | 0.5 | 11.23 | 78.9 | 66.9 | 81.0 | | | ln3 (default) | 14.80 | 79.0 | 67.1 | 81.1 | | | 2.0 | 21.84 | **79.1** | **67.1** | **81.1** | | | 3.0 | 30.41 | 79.0 | 66.9 | 80.9 | | | 5.0 | 76.66 | 69.9 | 68.9 | 74.6 | | **DPO-D (beta=0.5)** | -2.5 | 11.56 | 79.0 | 67.0 | 81.0 | | | -1.0 | 15.25 | 79.0 | **67.1** | **81.1** | | | 0.0 (default) | 11.66 | **79.2** | 67.1 | 81.0 | | | 1.0 | 29.72 | 79.0 | 67.0 | 81.0 | | | 2.5 | 47.20 | **79.2** | **67.1** | **81.1** |
I would like to thank the authors for their responses. I will maintain my current evaluation.
Summary
The authors propose two methods to generalize DPO to accommodate tie (y_i = y_j). However, it may seem a minor problem from my perspective. DPO cannot accommodate tie but it can tolerate when y_i is very close to y_j. In addition, one problem is that the authors does not compare the proposed two method with DPO without considering tie. Will its performance be lower than DPO? The author utilize a relative subjective method to compare models, while classification and generation tasks should be done.
Strengths
The authors propose two methods to generalize DPO to accommodate tie (y_i = y_j).
Weaknesses
However, it may seem a minor problem from my perspective. DPO cannot accommodate tie but it can tolerate when y_i is very close to y_j. In addition, one problem is that the authors does not compare the proposed two method with DPO without considering tie. Will its performance be lower than DPO? The author utilize a relative subjective method to compare models, while classification and generation tasks should be done.
Questions
NA
Rating
6
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Thanks for your review. In response to your concerns: > However, it may seem a minor problem from my perspective. DPO cannot accommodate tie but it can tolerate when y_i is very close to y_j. Thanks for raising this. As we show in the paper, including closely-matched pairs in DPO, i.e., DPO(CP+TP), hurts performance substantially (Figure 1 & 3). This identifies a weakness that DPO degrades in performance when closely matched pairs are included in training. In our work, we propose two tie-compatible DPO variants that address this problem. Our results suggest that by explicitly modeling for the possibility of a tie, DPO-RK and DPO-D achieve similar performance as DPO trained on Clear Preference only (i.e., DPO(CP)) when closely matched pairs are included as Tied Pairs. Additionally, DPO-RK and DPO-D provide stronger regularization in terms of KL divergence and thereby achieving more efficient KL-Performance frontiers. > one problem is that the authors does not compare the proposed two method with DPO without considering tie. Will its performance be lower than DPO? Our algorithms are intended for the case when the training data contains ties. If the dataset doesn't contain ties, DPO and any other DPO variants can be used directly. We note that we report the performance of DPO without ties for all our experiments (Blue curves in Figure 1 and Figure 3). We refer to this configuration as DPO(CP) where CP stands for Clear Preference. We find that DPO(CP) achieves better performance than DPO(CP+TP) on all three experimental setups. > The author utilize a relative subjective method to compare models, while classification and generation tasks should be done. We respectfully disagree. We report results on Neural Machine Translation (NMT) and summarization. NMT is a well-studied generation task with trusted and carefully calibrated automatic metrics. We use BLEURT [6] and COMET [7, 8] and have added KIWI-22 [4] (please refer to response to Reviewer kSzv for detail). The WMT23 metric overview paper [3] shows that these metrics are among the best available for this task. For summarization, we use TL;DR which was one of the tasks studied for the original DPO paper. We evaluate with win-rate against 256 human-written summaries as judged by PairRM which we demonstrate to follow closely to the win-rate as judged by GPT-4 (Appendix C.3 in our paper). We further evaluate the best-performing variant under each configuration with GPT-4 to strengthen our evaluation (please see response to reviewer ehu6 for detail) which we will include in revision.
Summary
The paper incorporates the pair-wise ties into DPO alignment framework, by replacing the BT model in DPO with Rao and Kupper and Davidson extensions. It conducted performance on machine translation and summarization tasks. Also, it provides some analysis on KL divergence and RM statistics. It provides potential insights to incorporating tie pairs for DPO training.
Strengths
1. The motivation is straight forward and easy to follow. It provides the straightforward idea that using the discarded tie pairs for optimizing the DPO framework. 2. The injection of DPO-RK and DPO-D remains similar task performance in comparison to original DPO, mitigating the problem of original DPO that will suffer from task performance degradation when injecting tied pairs. 3. The background is well presented and in a good shape.
Weaknesses
1. The paper experiments is not quite sufficient. It only conducted experiments on machine translation and summarization, rather than conducting commonly used alignment datasets such as HH-RLHF (https://huggingface.co/datasets/Anthropic/hh-rlhf). 2. Limited evaluation metrics. It only uses automatic evaluation metrics BLEURT and RM win rate, which is not often in consistent with human alignments. It is required to further report the results of gpt-4 evaluation and human evaluations. 3. It is recommended to report the case to better illustrate the effect of proposed methods. 4. The methods of obtaining tie pairs requires further evaluation. Is the automatic distance-based metrics reliable to obtain tie pairs? It is recommended to report the evaluation results.
Questions
1. What is the human evaluation results on the DPO-RK/DPO-D results compared to original DPO? 2. How is the results on commonly used benchmarks, such as HH-RLHF?
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Thank you for your review and suggestions. In response to your questions and concerns: > It only conducted experiments on machine translation and summarization, rather than conducting commonly used alignment datasets such as HH-RLHF (https://huggingface.co/datasets/Anthropic/hh-rlhf). Thanks for raising this. Most of the commonly used datasets do not contain ties and so they do not provide the information that is needed for our experiments. So, in order to conduct our experiments with ties, we need both (1) multiple responses to a given prompt, and (2) reliable automatic metrics to determine which pairs are ties. The TL;DR dataset on summarization has multiple responses for a given prompt and so we can select ties from these in an unsupervised manner. The WMT dataset on translation does not contain multiple translations for a source sentence, but these can be readily generated using open-sourced models and ranked with well-studied automatic metrics in machine translation literature, as done in [1]. HH-RLHF, however, does not have multiple responses per prompt and neither do we have metrics that are as reliable and well-studied as the ones for machine translation. Therefore, we did not choose this dataset for the study of ties. We hope this work could encourage people to keep ties in the dataset as opposed to discarding them. > It is required to further report the results of gpt-4 evaluation and human evaluations. Thanks for raising this. For NMT, we make use of automatic metrics that have been extensively studied and shown to correlate well with human judgment on translations [3 Table1]. We report BLEURT and COMET scores in Table 5, 6 in Appendix C. Recent published work on NMT with preference optimization [1, 2, 5] have used them as main evaluation metrics. On NMT, we additionally report KIWI-22 [4], a recent reference-less neural metric for translations in our response to Reviewer kSzv. For TL;DR, we additionally report the win-rate as evaluated by GPT-4 of the best performing systems below. | Model | Best PairRM Winrate | GPT-4 Winrate | |------------------------------|---------------------|---------------| | Pythia-2.8-DPO(CP) | **71.5%** | 62.1% | | Pythia-2.8-DPO-D(CP+TP) | **71.5%** | **64.1%** | | Pythia-2.8-DPO(CP+TP) | 69.5% | 62.1% | | Pythia-2.8-DPO-RK(CP+TP) | 68.0% | 57.8% | | Pythia-2.8-SFT | 37.5% | 36.7% | We note that the rank based on PairRM win-rate agrees with the rank produced by GPT-4 except that GPT-4 prefers Pythia-2.8-DPO-D(CP+TP). Thank you for your suggestion. We will include these results in the Appendix to further strengthen our experimental results.
> It is recommended to report the case to better illustrate the effect of proposed methods. Thanks for raising this. We collect the responses on WMT-21 ZH-EN from the best performing model under each configuration. Below, we present two cases where systems trained with DPO-RK(CP+TP) and DPO-D(CP+TP) perform better than DPO(CP) and one case where they perform worse than DPO(CP) as judged by KIWI-22. We will include more cases in the Appendix. The KIWI-22 score of each translation is shown in parentheses. ID: 10 **Source**: 小康不小康，关键看老乡。 **Reference**: Hometowns are the key to moderate prosperity. **DPO(CP)**: It is not important whether you are rich or poor, as long as you have a good relationship with your neighbors. (71.6) **DPO(CP+TP)**: Wealthy or poor, it all depends on our neighbors. (72.0) **DPO-RK(CP+TP)**: Wealthy or poor, it all depends on our neighbors. (72.0) **DPO-D(CP+TP)**: Whether you are rich or poor depends on your neighbors. (77.9) In this case, the DPO(CP) system produces a long translation with incorrect meaning. The other three systems produce correct, literal translations. The translation from DPO-D(CP+TP) is more fluent. However, all systems fail to fully capture the meaning of "小康" (i.e.,"moderate prosperity") as done in the reference. ID=269 **Source**: 这一航空运营商在声明中说，旗下一架客机7月15日从华盛顿州西雅图飞往得克萨斯州奥斯汀途中，一台发动机突然熄火，客机最终在奥斯汀机场成功紧急降落。 **Reference**: The airline released a statement claiming that as its aircraft was flying from Seattle, Washington to Austin, Texas, one of the engines suddenly stalled, and the aircraft finally made an emergency landing in Austin. **DPO(CP)**: This airline operator said in a statement that one engine on one of its aircraft failed midflight on July 15 from Seattle, Washington to Austin, Texas, and the aircraft successfully made an emergency landing at the Austin airport. (84.5) **DPO(CP+TP)**: This airline operator said in a statement that one of its aircraft was on its way from Seattle, Washington to Austin, Texas when one of its engines suddenly went out of order, and the aircraft successfully made an emergency landing at the airport in Austin. (81.4) **DPO-RK(CP+TP)**: This airline operator said in a statement that one engine on one of its aircraft failed during the flight from Seattle, Washington to Austin, Texas on July 15, and the aircraft successfully made an emergency landing at the Austin airport.(85.4) **DPO-D(CP+TP)**: This airline operator said in a statement that one engine on one of its aircraft failed during the flight from Seattle, Washington to Austin, Texas on July 15, and the aircraft successfully made an emergency landing at the Austin airport. (85.4) The translation produced by the DPO(CP+TP) system fails to capture the date of the event ("July 15"). Translations from DPO-RK(CP+TP) and DPO-D(CP+TP) are identical and differ from that from DPO(CP) only in rephrasing "midflight" with "during the flight". ID=91 **Source**: 运动少年热血竞技秀《运动吧少年》在7月11日正式登陆湖南卫视。 **Reference**: The inspiring competitive show for youth sports, We Are The One, was officially broadcast on Hunan TV on July 11. **DPO(CP)**: The youth sports action show “Be Active, Be Sporty” will officially debut on Hunan Satellite TV on July 11. (76.8) **DPO(CP+TP)**: The youth sports competition show "Go, Boy!" officially landed on the Changsha Satellite TV Station on July 11. (75.6) **DPO-RK(CP+TP)**: The youth sports competition show "Go Play Boys" will be officially broadcast on the Hunan Satellite TV on July 11. (72.1) **DPO-D(CP+TP)**: The youth sports competition show "Let's Go Boys" will officially debut on the Hunan Satellite TV on July 11. (74.2) In this case, all systems produced a more literal translation of the name of the TV show compared to the reference. Although the KIWI-22 scores for DPO-RK(CP+TP) and DPO-D(CP+TP) are lower, the translation qualities are in fact similar.
> Is the automatic distance-based metrics reliable to obtain tie pairs? Thanks for raising this. We conduct the following experiments with GPT-4 to evaluate whether the tied pairs selected by BLEURT are reliable. We randomly select 128 source sentences on our training set sampled from WMT20 ZH-EN and collect their corresponding CPs and TPs as determined by BLERUT score's ranking. Then, for each source sentence, we present the two pairs of translations to GPT-4o-2024-08-06 and prompt it to decide which pair of translations is a tied pair. We find that on the 128 randomly selected examples from WMT20 ZH-EN, the GPT-4o model agrees 82.03% of the time with BLEURT's assignment of CP and TP. We attach an example of our prompt and GPT-4's response below for your reference. In this case, GPT-4 chooses the same pair (Pair A) as TP as BLEURT. We will attach these results to the Appendix in our revision. We also note that the WMT23 metric reports high correlation (0.776, Table 1 in [3]) between BLEURT score and human judgment on translation. ```Source Sentence: 北京青年报记者了解到，首次公布的《意见稿》有多处重要修改。=. Pair A: Better Translation: Beijing Youth Daily reporters learned that there were many important amendments in the first published " drafts". Worse Translation: Beijing Youth Daily reporters learned that the "First Draft" has many important modifications. Pair B: Better Translation: The Beijing Youth Daily reporter learned that there were many important changes in the Draft for the first time published. Worse Translation: Beijing Youth Daily reporter learned that the first version of the "Comments" had been revised many times. You should provide a short rationale and then decide which pair is a tied pair. You should follow the format "DECISION: [[A]]" or "DECISION: [[B]] " to give your decision. Response: To determine which pair represents a tied pair, let's evaluate the differences between the translations in each pair. In Pair A: - The "better" translation is: "Beijing Youth Daily reporters learned that there were many important amendments in the first published 'drafts'." - The "worse" translation is: "Beijing Youth Daily reporters learned that the 'First Draft' has many important modifications." Both translations in Pair A convey similar meanings, describing modifications or amendments to a draft. The primary differences are stylistic, such as the placement and quotation of "draft". In Pair B: - The "better" translation is: "The Beijing Youth Daily reporter learned that there were many important changes in the Draft for the first time published." - The "worse" translation is: "Beijing Youth Daily reporter learned that the first version of the 'Comments' had been revised many times." In Pair B, the two translations have significant differences in meaning. The "better" translation accurately captures the notion of "important changes" and "first published," whereas the "worse" translation introduces "first version of the 'Comments'" and a focus on revisions rather than initial publication, which diverges more from the source sentence. With this analysis, Pair A has translations that are more similar in meaning and are thus likely to be the tied pair, while Pair B shows a clearer distinction in translation quality. DECISION: [[A]] ```
Thank you for reporting the qualitative cases and win rates on TL;DR. As mentioned by the authors, HH-RLHF did not contain the tied pairs so that they did not conducted fair comparisons with commonly used RLHF benchmarks, such as HH-RLHF, WebGPT Comparisons, etc. However, no existing RLHF benchmarks contain the tied pairs, posing great challenges in applying the proposed methods in practice. Again, it lacks sufficient evidence showing the claimed performance gains on commonly used RLHF datasets. Overall, I maintain my previous assessment in light of the author's update.
> However, no existing RLHF benchmarks contain the tied pairs, posing great challenges in applying the proposed methods in practice. Thanks for raising this. We note that one reason that existing RLHF benchmarks do not contain tied pairs is that there are no preference optimization algorithms that explicitly accommodate them. As we have shown in the paper, the Bradley-Terry-based DPO in fact degrades from including tied pairs in training. In this paper, we show that tied pairs could provide regularization and that our DPO-RK(CP+TP) and DPO-D(CP+TP) systems achieve similar performance as DPO(CP) systems on translation and summarization. This is achieved at lower KL and without the performance degradation seen in DPO(CP+TP). We hope that these findings and our proposed DPO variants would encourage developers to keep closely matched pairs in future release as opposed to simply discarding them. > Again, it lacks sufficient evidence showing the claimed performance gains on commonly used RLHF datasets To clarify, we did not claim performance gains on commonly used RLHF dataset. The aim of the paper is to study DPO variants derived from tie-compatible extensions of Bradley-Terry. For this purpose, we conducted experiments on machine translation and summarization where ties can be identified via reliable automatic metrics or in an unsupervised manner provided multiple responses to a prompt.
Summary
This paper introduces two variants of Direct Preference Optimization (DPO) designed to account for ties in pairwise preference data, addressing a limitation in the original DPO framework that requires strict winner-loser pairs. The authors replace the Bradley-Terry model traditionally used in DPO with two extensions from Rao-Kupper and Davidson models, both of which assign probabilities to ties in addition to clear preferences (and have a nice explanation of the different assumptions in each). The proposed DPO variants—DPO-RK and DPO-D—are then tested on neural machine translation and summarization tasks, using synthetic preference pairs (with ties) constructed metrics/reward models. The experiments show that incorporating ties into DPO-RK/D does not degrade task performance as it does in traditional DPO and, in fact, regularizes the policy with respect to the reference policy, as seen through KL divergence measurements. They also test the learned policies’ classification accuracy at distinguishing between clear preferences and ties.
Strengths
- Overall, I like the paper’s motivation and proposed approach: DPO is very prevalent method to perform preference optimization, and the inability to handle ties is a definitely a drawback. - The proposed variations are well motivated and the theoretical discussion on their differences was quite interesting to me. - The (limited) results included hint that including ties and using one their proposed variants leads to better policies than the current dominant approach of discard ties.
Weaknesses
While overall like the idea and introduced approach, I think the paper is severely limited by the weak empirical evaluation. A couple of points: - Reporting of results for both task are relatively limited, only reporting performance for preference aligned models on the metrics used to construct preference. I think having a table reporting baseline and best performing DPO models on multiple metrics is necessary to trust gains from DPO and its variants, as they could be over-optimizing for BLEURT. In MT for example, gains in COMET (or even BLEU/ChrF) would allow me to trust to overall conclusions alot more. - For machine translation, a relatively weak model (BLOOMZ-mt-7b) is used. Why this model rather than many strong LLM for MT e.g. TowerLLM? Similarly, BLEURT is relatively outdated as a metric for MT, and I would have used a better metrics to construct preference pairs. In general, replicating the findings in more than one base LLM (for at least MT) would strengthen the paper. - In general, there lack of comparison to some other baseline preference alignment algorithms (KTO, IPO, etc…). I would really like to see how DPO-RK/D compares one of them, especially if tie data is also used with it. But in general I like the motivation of the paper, and would be happy to revise my score if authors address these comments
Questions
- What happens when the proportion of ties vs winning pairs changes (in terms of performance, KL, etc…)? Since the preference pairs are synthetic, this would be something relatively easy to explore, and could inform what kinds of preference datasets make DPO-RK/D work well.
Rating
6
Confidence
3
Soundness
2
Presentation
3
Contribution
3
Thanks for your knowledgeable review and positive comments on our methodology. In response to your concerns: > I think having a table reporting baseline and best performing DPO models on multiple metrics is necessary to trust gains from DPO and its variants Thanks for raising this point. We note that BLEURT and COMET scores for our NMT experiments are given in Table 5 and 6 in Appendix C. We additionally evaluate our systems with KIWI-22 [4], a popular reference-less neural metric used by recent works in evaluating SoTA machine translation systems [2]. We will add the following table in the Appendix in our revision. | Model | Best BLEURT | Best COMET | Best KIWI-22 | |------------------------------------|-------------|------------|--------------| | Bloomz-mt-7b1-SFT | 61.6 | 77.9 | 77.3 | | Bloomz-mt-7b1-DPO (CP) | **67.1** | **81.3** | **82.0** | | Bloomz-mt-7b1-DPO (CP+TP) | 66.6 | 80.6 | 78.5 | | Bloomz-mt-7b1-DPO-RK (CP+TP) | **67.1** | 81.0 | 79.0 | | Bloomz-mt-7b1-DPO-D (CP+TP) | **67.1** | 81.0 | 79.2 | | Bloomz-mt-7b1-IPO (CP) | 66.6 | 80.6 | 78.6 | | Bloomz-mt-7b1-IPO (CP+TP) | 66.3 | 80.5 | 78.4 | This shows clear gains from DPO and its variants compared to the SFT baseline under three metrics, two reference-based and one reference-less. We note that these are all strongly rated metrics in the WMT-23 metric overview paper [3, Table 1] > there lack of comparison to some other baseline preference alignment algorithms (KTO, IPO, etc…). I would really like to see how DPO-RK/D compares one of them, especially if tie data is also used with it. Thanks for raising this. We agree that comparison with other alignment algorithms can strengthen the paper. We perform additional NMT experiments with IPO fine-tuning on CPs (IPO(CP+TP) and CPs+TPs (IPO(CP+TP)). We conduct IPO fine-tuning (using code from https://github.com/eric-mitchell/direct-preference-optimization) with the a range of beta values [0.025, 0.05, 0.1, 0.3, 0.5, 0.7, 1.0] and report the best performing system as evaluated on the WMT-21 ZH-EN dataset below. We use identical training hyper-parameters as used in DPO, DPO-RK and DPO-D fine-tuning. Please refer to the table in the response above. We find that 1. both IPO(CP) and IPO(CP+TP) yield substantial gains compared to the baseline SFT model (+4.7 BLEURT); 2. Adding TPs to IPO causes a small degradation (<-0.3) across all metrics; 3. IPO(CP) performs on par with DPO(CP+TP) but under-performs DPO(CP), DPO-RK(CP+TP), and DPO-D(CP+TP) on all metrics. Thank you for the suggestion. We will include these results in the Appendix. > as they could be over-optimizing for BLEURT. ... gains in COMET (or even BLEU/ChrF) would allow me to trust to overall conclusions alot more. Thanks for raising this insightful point. Indeed, we see instances of systems over-optimizing BLEURT in our experiments, which leads to degradation in COMET scores compared to the baseline model. We discussed this issue of "reward hacking" in the paper (Line 1132-1145) which we reproduce below for your convenience. In short, we find that increasing regularization with respect to the reference model, either by using a larger beta value or incorporating ties in DPO-RK/DPO-D training, can resolve the problem. >> "We observe the ``reward hacking'' phenomenon identified by Yang et al. (2024b) on both WMT21 ZH-EN and IWSLT17 FR-EN where systems achieve good BLEURT but have large length ratio (>1.5) and lower COMET than the pre-DPO system. These systems learn to generate long, repetitive translations which BLEURT fails to recognize as low-quality. Yang et al. (2024b) find that using small beta values (e.g. 0.1) in DPO training results in reward hacking models. Our results are consistent with their findings and further suggest that large KL divergence from the reference model is a good indicator for reward hacking. On WMT21 ZH-EN, the only model that exhibits reward hacking is trained by DPO(CP) with beta=0.1 which also yields the highest KL divergence (174.13) among all models, greatly exceeding the second-highest KL divergence (68.12). On IWSLT17 FR-EN, Almost all models with KL Divergence > 30 (DPO(CP), $\beta=0.1$, DPO-RK(CP+TP), $\beta=0.1$ and DPO-D(CP+TP) $\beta=0.1, 0.5$) show reward hacking behaviours. Reward hacking on NMT can be resolved by increasing regularization with respect to the reference model. We find that training with larger beta values or incorporating ties in DPO-RK/DPO-D training can provide such regularization without performance degradation."
> For machine translation, a relatively weak model (BLOOMZ-mt-7b) is used. Why this model rather than many strong LLM for MT e.g. TowerLLM? Thanks for raising this. We use BLOOMZ-mt-7b because it has been shown to improve substantially from DPO fine-tuning on self-generated preference pairs [1]. This allows us to synthesize ties easily and reliably and to compare our tie-compatible DPO variants against a strong DPO baseline. Given our goal to study tie-compatible extensions of DPO rather than SoTA translation model, we find the published BLOOMZ-mt-7b setup to exhibit the type of behavior we wish to study. We are running additional experiments using the more performant ALMA-7B-LoRA model as reported in [2]; Please see response below for detail > BLEURT is relatively outdated as a metric for MT, and I would have used a better metrics to construct preference pairs Thanks for raising this. We believe BLEURT is sufficient for our purpose of selecting CPs and TPs in that the quality difference between translations in CPs (i.e., translations with high BLEURT difference) is substantially greater than that in TPs (i.e., translations with small BLEURT difference). This is supported by the WMT23 metric paper which reports high correlation (0.776, Table 1 in [3]) between BLEURT score and human judgment. We additionally evaluate whether GPT-4 agrees with BLEURT's CP/TP assignment on the training set given the two pairs of translations and find that GPT-4 agrees 82% of the time (please see response to Reviewer ehu6 for detail). We are replicating our findings with the ALMA-7B-LoRA model (see below) where we use XCOMET to construct preference pairs. > In general, replicating the findings in more than one base LLM (for at least MT) would strengthen the paper. Thanks for your suggestion. Following your advice, we are conducting additional experiments with ALMA-7B-LoRA, a recent strong-performing LLM specialized for translation. We follow the experimental setup in [2] where a triplet preference dataset is available. The dataset provides three translations for each source sentence as ranked by the average of KIWI-XXL [12] and XCOMET [9] scores. [2] only uses the two translations with the highest score margin in preference optimization. We additional include the two translations with minimal score difference as Tied Pairs (TPs) and train with DPO, DPO-RK, and DPO-D. We sweep beta in the range of [0.1, 0.2, 0.3, 0.5, 0.7] and train on the ZH-EN and EN-ZH directions only and evaluate on the ZH-EN direction. Following [2], we evaluate with KIWI-22, KIWI-XXL, and XCOMET These metrics show high correlation with human judgment in the WMT23 metric over view paper [3]. | Model | COMET | KIWI-22 | KIWI-XXL | XCOMET | |----------------------------|-------|---------|----------|-------------| | ALMA-7B-LoRA (baseline) [2] | 79.8 | 76.8 | 73.7 |83.9| | ALMA-7B-R [2] | 81.0 | **78.5** | **75.7** |**90.7**| | DPO (CP) | 80.5 | **78.5** | **75.7** |90.3| | DPO (CP+TP) | 79.7 | 77.5 | 74.5 |88.9| So far, we have observed a similar degradation from including more closely-matched pairs to DPO in this published dataset with a strong base model. We are confident that our findings on BLOOMZ-mt-7B will carry over and are running experiments with DPO-RK and DPO-D. These results will be included in revision. However, as this involves training the system with a range of beta values, we are unsure if the full results will be available by the end of the discussion period. We will post results to this thread as they come up.
> What happens when the proportion of ties vs winning pairs changes Thanks for the suggestion. Following your advice, we conduct additional NMT experiments varying the proportion of Tied Pairs (TPs) and evaluate on the WMT-21 ZHEN dataset. We keep the Clear Preferences (CPs) unchanged and change the proportion of TPs kept in the training data. A TP% of 50% means that we keep 50% of all the TPs and so CPs:TPs = 2:1. For each tie proportion, we train systems with three beta values [0.2, 0.4, 0.6] and report the best performing system in terms of BLEURT. We report KL, KIWI-22, BLEURT, and COMET scores of these systems below. | Model | TP% | KIWI-22 | BLEURT | COMET | |-------------------|-------|---------|--------|-------| | DPO (CP) | 0% | **82.0** | **67.1** | **81.3** | | DPO (CP+TP) | 100% | 78.5 | 66.6 | 80.6 | | DPO-RK (CP+TP) | 25% | 78.4 | 66.5 | 80.7 | | | 50% | 78.4 | 66.2 | 80.7 | | | 75% | 78.5 | 66.6 | 80.8 | | | 100% | 79.0 | **67.1** | **81.0** | | DPO-D (CP+TP) | 25% | 78.5 | 66.4 | 80.8 | | | 50% | 78.5 | 66.3 | 80.7 | | | 75% | 78.6 | 66.5 | 80.8 | | | 100% | **79.2** | **67.1** | **81.0** | We find that DPO-RK and DPO-D achieve best performance when the number of TPs are balanced with the number of CPs. We also find that, under the same beta value, including more TPs reduce the KL divergence with respect to reference model at the end of training. This provides further empirical results for our analysis in Sec.3.1.1 that ties serve to regularize training. The following table reports DPO-RK(CP+TP) and DPO-D(CP+TP) systems with beta=0.4. | Model | TP% | KL | KIWI-22 | BLEURT | COMET | |-------------------|-------|-------|---------|--------|-------| | DPO-RK (CP+TP) | 25% | 10.16 | 78.4 | 66.5 | 80.7 | | | 50% | 6.05 | 78.2 | 66.2 | 80.4 | | | 75% | 4.40 | 78.5 | 66.4 | 80.7 | | DPO-D (CP+TP) | 25% | 17.26 | 78.5 | 66.4 | 80.8 | | | 50% | 10.40 | 78.5 | 66.3 | 80.7 | | | 75% | 6.63 | 78.5 | 66.5 | 80.8 | We will include these results in our revision. Thank you for your review. Your suggestions have been invaluable in enhancing the quality of our paper.
Reponse to Rebutal
I would like to thank the authors for their response and additional experiments. The additional metrics do alleviate some of the concerns with metric overfitting, but raise some questions about the benefit of the ties: DPO (CP) seems to get the same or even better results than DPO-RK/D (CP+T), which raises the question of what is the benefit of the later. Can you make a clear, concise argument for the later? Nevertheless, I'm raising my score slightly.
Thanks for acknowleding our work and for the great question. > The additional metrics do alleviate some of the concerns with metric overfitting, but raise some questions about the benefit of the ties: DPO (CP) seems to get the same or even better results than DPO-RK/D (CP+T), which raises the question of what is the benefit of the later. Can you make a clear, concise argument for the later? In response, DPO-RK/D training with tied pairs included in training can be useful when: 1. Ties are provided in the pair-wise dataset and it is of interest to learn from them. We have shown that DPO cannot learn well from tied data whereas DPO-RK/D can. 2. It is desirable to keep the trained policy relatively close to the reference policy in terms of KL divergence but without sacrificing performance. 3. The trained policy is intended to be used as a reward model to identify ties and clear preferences on held-out data. We have shown that DPO-RK/D(CP+TP) improves preference pair classification accuracy substantially compared to DPO(CP) and DPO(CP+TP). With regard to performance, we note that in summarization, we find DPO-D(CP+TP) to perform better than DPO(CP) by +2% in GPT-4 win-rate (please see response to reviewer ehu6 for detail). In NMT experiments with ALMA-7B-LoRA as the base model, we find a +0.2 gain from DPO-D(CP+TP) in terms of the mean of KIWI-XXL and XCOMET (please see below for detail) compared to DPO(CP). We hope this addresses your question and thanks again for your knowledgable review.
Additional NMT experiments with ALMA-7B-LoRA base model
Following your advice, we have conducted additional experiments, replacing the BLOOMZ-mt-7B model with the ALMA-7B-LoRA reported in [2] which has better translation performance. We repeat the procedure as described in Sec.3.1 to collect 3000 CPs and 3000 TPs for ZH-EN translation. This is done via sampling candidate translations using ALMA-7B-LoRA on the ZH-EN training set of [2] followed by pair selection based on scores from automatic metrics. Following [2], we use the average score of KIWI-XXL and XCOMET to rank responses. We evaluate our models on the ZH-EN test set of [2] which consists of 2009 parallel sentences selected from FLORES-200. We report COMET, KIWI-XXL and XCOMET as evaluation metrics. We also report KL divergence with respect to the reference model. The table below shows the best performing system as evaluated on ZH-EN translation obtained from a beta sweep in [0.1, 0.3, 0.5, 0.7, 0.9] under each configuration. We provide the performance of the base ALMA-7B-LoRA model as reported in [2] for reference. ALMA-7B-R is the best 7B model reported in [2] trained with Contrasitive Preference Optimization (CPO) on a larger dataset than what we used and so is not directly comparable. However, it provides a strong translation baseline for comparison. | Model | KL | COMET | KIWI-XXL | XCOMET | Mean(XCOMET+KIWI-XXL) | |-----------------------|-------|-------|----------|--------|-----------------------| | ALMA-7B-LoRA [2] | - | 79.8 | 73.7 | 83.9 | 78.8 | | | ALMA-7B-R [2] | - | 81.0 | 75.7 | 90.7 | 83.2 | | | **Our systems** | | | | | | | DPO(CP) | 101.64| **80.5** | 75.2 | **91.1** | 83.2 | | DPO(CP+TP) | 60.37 | 80.2 | 75.0 | 90.8 | 82.9 | | DPO-RK(CP+TP) | 75.57 | 80.2 | 75.0 | 90.8 | 82.9 | | DPO-D(CP+TP) | 30.92 | **80.5** | **75.9** | 90.8 | **83.4** | We get the best performance under the metrics with DPO-D(CP+TP) at the lowest KL on top of a very strong baseline. Consistent with all our other results, including Tied Pairs to all training procedures yields a strong regularization effect. Thanks again for your suggestion. We believe this set of additional experiments further strengthens our results.
References for Author Response
Please kindly see below for references used in authors' responses. [1] Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding (https://arxiv.org/abs/2311.08380) [2] Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (https://arxiv.org/abs/2401.08417) [3] Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent (https://aclanthology.org/2023.wmt-1.51.pdf) [4] CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task (https://aclanthology.org/2022.wmt-1.60.pdf) [5] A Preference-driven Paradigm for Enhanced Translation with Large Language Models (https://aclanthology.org/2024.naacl-long.186.pdf) [6] BLEURT: Learning robust metrics for text generation. (https://aclanthology.org/2020.acl-main.704/) [7] COMET: A neural framework for MT evaluation. (https://aclanthology.org/2022.wmt-1.52/) [8] COMET-22: Unbabel-IST 2022 submission for the metrics shared task. (https://aclanthology.org/2022.wmt-1.52/) [9] xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection (https://arxiv.org/abs/2310.10482) [10] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (https://arxiv.org/abs/2309.11674) [11] Tower: An Open Multilingual Large Language Model for Translation-Related Tasks (https://arxiv.org/abs/2402.17733) [12] Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task (https://aclanthology.org/2023.wmt-1.73.pdf)
Meta-review
This paper extends the Direct Preference Optimization (DPO) framework by introducing two variants, DPO-RK and DPO-D, which incorporate tied preferences in pairwise data using existings models (Rao-Kupper and Davidson). Through experiments on neural machine translation and summarization tasks, this study shows that these variants preserve more information by accounting for ties, offering improved regularization effects while maintaining task performance. Additionally, the paper provides analyses on KL divergence and tie classification, highlighting the potential benefits of integrating tie-aware models into DPO. Strengths: * The paper addresses a clear limitation of the DPO framework, where tied preference pairs are discarded, by proposing two extensions (DPO-RK and DPO-D) that leverage ties effectively. * The theoretical discussion and integration of Rao-Kupper and Davidson models provide a well-motivated foundation for handling ties in preference optimization. * The paper is well-written, with clear mathematical formulations and derivations, making the proposed methods easy to follow. * The experimental analysis, while limited, offers some insight into the performance improvements when incorporating tied pairs. Weaknesses: * The paper introduces methods based on existing extensions of the Bradley-Terry model, such as the Rao-Kupper and Davidson variants, but these modifications represent incremental advancements rather than fundamentally novel methodologies. * The experimental validation is limited in scope, focusing solely on machine translation and summarization tasks without exploring broader, more challenging domains like open-ended text generation or mathematical reasoning. * The work relies mostly on outdated or limited evaluation metrics. While the authors addressed some of the issues raised in the original reviews, the reviewers remain mostly unconvinced about the novelty and the level of difficulty of the studied tasks, even after considering the authors' responses. I share the reviewers' concerns and recommend rejection.
Additional comments on reviewer discussion
During the author-reviewer discussion, the authors provided numerous new results that strengthened the paper, addressing concerns about the evaluation metrics. This led one reviewer to increase their rating of the paper. However, during the reviewer-AC discussion, Reviewers T7fN and kSzv remained concerned about the limited novelty of this work, as adapting DPO by substituting the Bradley-Terry model with existing extensions appears incremental. Additionally, they found the experimental scope insufficient, with more challenging tasks left unexplored even after the authors' rebuttal. Their conclusion is that the submission does not meet the standards of ICLR.