SimPO: Simple Preference Optimization with a Reference-Free Reward

Summary

This paper presents SimPO, an offline preference optimization method for LLM alignment. SimPO replaces the KL term in DPO with a length-regularized log-probability and adds a margin value for regularization. Extensive experiments in chat benchmarks show that SimPO significantly outperforms DPO and other preference optimization variants, despite its simplicity. The authors conclude that the success of SimPO can be attributed to the better alignment between training and decoding objectives, as well as reduced exploitation of generating lengthy responses.

Strengths

+ This paper is well-written and easy to follow. The motivation is clear, and the main approach is quite understandable. + The experiments are extensive and well-designed. In addition to benchmark numbers, the authors also provide an in-depth analysis of generation lengths, log probabilities, and differences in rewards for DPO and SimPO models. This greatly helps readers understand the source of the performance gains.

Weaknesses

I do not find any significant weaknesses beyond the limitations proposed by the authors. Since I generally agree with these limitations, I am reiterating them in my review: + A theoretical understanding of SimPO is lacking. Although SimPO's design is not theoretically grounded, it performs well in practice, as validated by the authors' experiments. + The experiments in the paper solely focus on evaluating helpfulness, disregarding safety, honesty, etc. I think this is the major weakness. SimPO removes the KL regularization from the reference model, so it might intuitively suffer from safety issues. This problem should be further studied in the paper.

Questions

+ Could the authors explain why length-controlled WR is important? From my perspective, controlling response length is helpful during training because it encourages the model to generate short but useful responses. However, forcing the generation of short responses during evaluation is not necessary. Good responses are good regardless of length. Evaluating length-controlled WR seems to set a privileged metric for SimPO. + Since SimPO removes the KL regularization, how does it perform with poor data quality, which is a common application scenario? Is it more fragile and sensitive to data quality?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations have been addressed by the authors in Section 6 and in my weaknesses section.

Summary

This paper proposes a new offline-RLHF algorithm SimPO, which significantly improves current DPO variants on a collection of benchmarks.

Strengths

The SimPO algorithm is intuitive, simple to implement, and works well, with a good presentation. A few concrete strong points are list below. 1. The algorithm is intuitive and the overall design is novel in the current literature. The paper proposes two simple techniques, including directly optimizing the generation probability and adding a discrepancy term. Both two techniques are intuitive. Overall, little efforts in offline RLHF methods have fully explored these two directions systematically. 2. The experiments are solid. This paper conducts experiments on a diverse collection of chat benchmarks. The results are significant. 3. The overall presentation is great and easy to follow.

Weaknesses

The following comments are for potential improvement of this work and future research directions. These are not complaints. 1. Some fine-grained case studies can be beneficial. The paper primarily focuses on quantitative evaluations. But it would be interesting to see concrete changes between DPO and SimPO, i.e., some representative responses generated by SimPO and DPO, to see what has changed. The LN experiments are great. But things can be better. 2. With the current formulation, the SimPO objective reminds me of those online RLHF methods. Assuming a -1/1 reward function, many online RLHF methods optimize similar objectives, i.e., direction optimizing the responses with a 1 reward response (winning responses) while reducing the reward of bad ones (losing responses). It will be interesting to see some discussions between SimPO and online RLHF methods. Also, many recent practices have observed that online DPO works greatly. It would be interesting to see if the same phenomena can be observed for SimPO, which looks much closer to online RLHF variants. This is a potential direction to make this paper a much stronger one. 3. Another possible direction to explore is about tasks other than chat, e.g., reasoning tasks like coding and maths. The evaluations in this paper are indeed sufficient as they are, but some discussions on a broad domain can be appreciated.

Questions

I do not have any specific questions. As a final comment, if possible, I personally like to see some concrete examples of responses generated by SimPO and DPO (beyond the response lengths) to have a better understanding of the actual improvement.

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Most limitations have been declared by the authors.

Summary

Simpo introduces a simplistic new method of alignment leveraging the log-probability of the sequence eliminating the need to leverage the SFT policy making the implementation computationally less complex. By introducing a notion of margin in the loss function, SimPO outperforms several existing alignment algorithms including DPO and variants without increasing response length. Overall, it provides an interesting light-weight alignment procedure with strong empirical benefits.

Strengths

1. The approach eliminates the need for the reference model which can be computationally heavy or might not be available in practice 2. The methods shows improved empirical performance over DPO and existing baselines through extensive experimentations. 3. The method provides an interesting way of regularizing the length and shows improved performance without increasing the length of the response.

Weaknesses

1. The primary definition of RLHF has a constraint to the SFT policy through the KL regularization which restricts the model to overoptimize the reward function. Also, it ensures the base performance of the LLM remains intact on several other domains and tasks. However, the proposed method doesn't use KL to the SFT and removes the SFT which makes it unclear what to attribute the improvement to. Specifically, one can violate the constraint and do better with the reward for that specific or similar task, but what about the performance in other tasks that the pre-trained/SFT model was good at? 2. DPO has a closed-form unique solution due to the strong convexity of the KL regularized RLHF objective resulting in the particular objective. However, it's not clear if we don't use the KL regularization, what the original optimization problem we are optimizing for. Is it still strongly convex? If not, what are the insights that this objective will have some good convergence properties? 3. Under all these questions, the motivation for the objective is heuristically driven and unclear from where it comes from

Questions

1. Can you mention if your original objective is strongly convex or how the particular objective is derived? 2. Can you provide the KL regularization of your model to the SFT and show how much it deviates from the SFT in comparison to baselines?

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Check above

Summary

The paper introduces SimPO (Simple Preference Optimization), an extension of Direct Preference Optimization (DPO), by replacing the reference-policy-dependent implicit reward with a reference-free reward. Specifically, SimPO utilizes the average log probability of a sequence as the implicit reward, aligning it with the generation process and eliminating the need for a reference model. This results in improved computational and memory efficiency. Additionally, a target reward margin is introduced into the Bradley-Terry objective to ensure a significant reward difference between winning and losing responses. Extensive experiments show that SimPO consistently outperforms DPO and its variants across various benchmarks, demonstrating its effectiveness in improving model performance.

Strengths

- The focus on addressing the discrepancy between reward and generation in DPO is novel and valuable, providing a new perspective on extending DPO. - SimPO enhances computational and memory efficiency by eliminating the need for a reference model. - Extensive experiments across various benchmarks demonstrate that SimPO consistently outperforms DPO and its variants.

Weaknesses

- Although SimPO simplifies the optimization process by removing the reference model calculation of response probabilities, it introduces a new hyperparameter, the target reward margin $\gamma$, which requires tuning. - While the motivation to address the discrepancy between reward and generation in DPO is understandable, there is a lack of mathematical discussion on how the proposed implicit reward is effective for preference optimization. For instance, standard random sampling with temperature 1 could eliminate the discrepancy without length normalization. However, the experimental results suggest that length normalization is crucial, questioning whether the initial motivation to align optimization and decoding is definitively effective for preference optimization. - The target reward margin plays a critical role in SimPO, but there is no discussion or comparative experiments with "DPO with an offset" (Amini et al., 2024), which also introduces a similar margin into the DPO loss. This comparison is necessary to understand the relative advantages of SimPO.

Questions

- Is it possible to add a careful discussion and conduct comparative experiments about DPO with a target reward margin (Amini et al., 2024)? - The paper suggests that reducing the discrepancy between reward and generation in DPO results in a more preferable reward for preference optimization. While theoretical analysis might be challenging, is it possible to experimentally validate this hypothesis? For instance, can you evaluate the relationship between the discrepancy's size and preference optimization's performance? This could include testing different decoding methods, such as a beam search with larger beam widths in the length-normalized version that will align more with the average log-likelihood reward. - Since SimPO omits KL divergence regularization from the initial policy, assessing how the KL divergence behaves would be helpful. Evaluating the model's performance in terms of KL divergence, as analyzed by Gao et al. (2023), could provide valuable insights.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately discussed the limitations.

Thank you for your response.

I'v read the authors' comments to all reviewers. I don't have any additional concerns and would like to remain my score, voting for acceptance.

Response to Rebuttal by Authors

Thanks for the points. I, in general, agree with the comment regarding the closed form of DPO and that it directly doesn't hold for a parametrized setting, which is ok. However, at least for the Tabular class of policies, it holds true and it indeed has the closed form. However, in your problem, how the optimization expression comes is not clear, once we remove the KL. Also, can you explain why even after not having any KL constraint, SimPO can achieve similar KL divergence to DPO ? Which hyperparameters will cause it to stay close and why?

Thanks

Thanks for the reply and I appreciate the authors' efforts.