Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Summary

This paper analyse overoptimisation in the context of direct alignment algorithms (DAA). They show that even when no explicit reward model is being optimised against, a similar phenomena as Gao et al. is shown, where as KL budget increases "gold" reward increases and then decreases. This phenomena is shown across model sizes and different DAAs (DPO, IPO, SLiC). They fit scaling curves to this phenomena and find a similar style of fit to Gao et al.. This phenomena is then analysed from various angles, and the authors show that at low KL budgets best performance is reached early in training; length-correct adjusts the KL-winrate pareto frontier but doesn't mitigate overoptimisation; model training statistics aren't useful for predicting downstream performance. They also present some theoretical analysis and a toy MDP which demonstrates why reward exploitation can occur in DAAs, showing that lack of support in the sequence space combined with DAA's non-convex optimisation target can lead to DAAs placing high probability on OOD (and hence low-reward) sequences. L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. Interna- tional Conference on machine Learning, 2023.

Strengths

The paper's analysis is interesting and novel - while overoptimisation has been demonstrated in online RL preference learning algorithms, this demonstration of the effect in DAAs hasn't been presented before to my knowledge. The quality of the analysis of the paper is high - they discuss their results neutrally and clearly, and the results are somewhat general. The additional analysis answers many obvious questions that come to mind, which is beneficial; the authors have clearly investigated this phenomena thoroughly. The paper is well-presented and easy to understand. The significance of the work is reasonably high - DAAs are a very popular class of preference learning algorithm, and this work has demonstrated a previously unknown limitation of these algorithms, and provided much additional analysis and understanding into how they work and where they can break, which will be useful for the community going forward in improving upon these algorithms and understanding their limitations.

Weaknesses

## Large points I think the main weaknesses with the paper are three-fold: * All analysis is only performed on the TL;DR summarisation dataset, which is somewhat different from Gao et al. and the setting these preference learning algorithms are generaly used in (dialogue and instruction-following). While reproducing the whole analysis on another dataset is too much to ask, testing out some of the core hypotheses, or reproducing some of figure 1, on a different dataset (e.g. alpaca farm, or anthropic HH) would increase the robustness and generality of the results. * Using GPT-4 winrate as the "gold" reward or reference output also makes the results less general. The preference distribution that produced the dataset the DAAs were trained on is not the same one being used to evaluate here. I would expect the results to hold up in the setting where these two distributions are the same, but it would be beneficial to have some empirical validation of that. This would likely mean mimicing the setting of Gao et al, and applying DPO on a preference distribution generated from an accessible gold reward function (for example, the data of Coste et al. is available and may be suitable, or alpaca farm gpt4 preference data could also work). The same reward function can then be used to produce the y-axis of the plots, which might even produce cleaner trends. This would more accurately mirror the real-world setting where human preferences would be used for both training and evaluation. * While the dicussion in section 4 is useful, it would be beneficial to have a better explanation of why this phenomena happens in DAAs, as it's still not immediately intuitive for me. ## Small points * it would be useful for the axes in figure 1 to be unified across all the subplots, so it is easier to compare between algorithms. Similarly for the y-axis in figure 2. * some of the text in the figures (especially figure 7) is quite small. ## Summary I think the paper is worthy of acceptance as is, and I'm reccomending accept. If one of the points mentioned in the Large Points section above was addressed thoroughly I would consider raising my score to a strong accept.

Questions

My questions have been described in the weaknesses section.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discuss limitations.

Summary

This paper studies the reward over-optimization issue for offline alignment algorithms (e.g., DPO series) with massive experiment trials and discussions on why this phenomenon happens.

Strengths

1. This is the first paper that studies the over-optimization issue for DPO-like algorithms systematically. The observations and findings can be beneficial to the community. The proposed experiments are organized rigorously with sufficient quantitative results. 2. The paper presents concrete examples to reveal why over-optimizations could happen for DPO-like algorithms, which brings insights to the community.

Weaknesses

There are no major weaknesses. Here are some of the minor comments for further improvement. 1. Why do you choose DPO, IPO, and SLiC-HF? Why not some other variants, say ORPO? Although I do agree that the current selections can be sufficiently representative, some discussions on why they are prioritized can be appreciated. 2. The presentation of Section 4 can be possibly improved. With its current presentation, it is difficult for a reader to understand the major argument of this section. There seems to be a collection of diverse arguments with different experiments. It may be better to highlight the overall goal of this section and the reason why these experiments are conducted in the very beginning before moving into more detailed experiments. 3. Regarding the "Rank Efficiency" issue in Section 4, is it possible to have some more rigorous formulations in addition to statements only, say a theorem? 4. Regarding citations and references, the OOD issue of DPO is also discussed in [1], which may be a good one to refer to. Also, the citation format is very strange. For example, many cited papers in the reference do not have any publication venues. [1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, S. Xu et al., ICML 2024. https://arxiv.org/abs/2404.10719

Questions

See the comments in the weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations have been discussed in the paper.

Summary

This paper studies the scaling laws of DAAs for RLHF. The authors conducted extensive empirical studies and the discoveries are reported and discussed.

Strengths

The paper is well-written and easy to follow. The authors conducted extensive empirical studies. The current results are useful to the community to understand different algorithms. The design of the diagnostic toy MDP is interesting and smart.

Weaknesses

I like large-scale empirical study yet I disagree with calling those discoveries "laws". To draw a clear scaling **law** (from a scientific perspective), the current results are not supportive enough. Specifically, in Figure 1, the results do not seem to be a good fit, and more experimental results might be helpful to draw the conclusion. Also, why is the form of "scaling law" chosen as Eqn(5)? In Gal et al, BoN and PPO have different forms --- it would be highly possible that different DAAs also have different equation forms. The authors should further highlight the takeaway messages of their empirical discoveries. For instance, with the discovered scaling law, is there a way to perform early stopping in training to achieve better performance? --- This could be a great contribution to the community as otherwise researchers may be prone to make unfair comparisons in evaluating algorithms.

Questions

Could the authors also provide error bars in Figure 2, second row? The results look very unstable. For the scaling law fit, the author mentioned using win-rate to be the proxy of the golden reward, could the author provide detailed statistics of the results? I'm curious --- to what extent could different judges agree? If the win rate is only accurate in (for example,) 80% of the settings, does not that mean we are only able to draw a very rough conclusion using another proxy of this objective? On the Tree-MDP, why should all states go to the same absorbing state? What is the consideration of using a single absorbing state rather than many (i.e., = the number of leaves).

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

please see weakness / question.

Summary

Reinforcement Learning from Human Feedback (RLHF) is a popular paradigm for aligning Large Language Models (LLMs) to human preferences. Direct Alignment Algorithms (DAA) are an alternative to traditional RLHF methods, which reduce the need to learn a reward and policy model separately. It has been shown that traditional RLHF methods suffer from reward over-optimization or reward hacking. However, because DAA do not learn a separate reward model, it is important to study whether these algorithms suffer from the same issues. This paper studies three popular DAA algorithms and how their performance deteriorates from over-optimization at scale. The paper has several compelling results that show the trade-off between different DAA objectives, KL divergence, and win rate when optimizing across different scales of models.

Strengths

- Empirical experiments provide insight into DAA algorithm's behavior at scale. - The empirical verification of the decreasing likelihood and model performance provides insight into an important issue in the literature: the decrease in the responses of both chosen and rejected samples. - The paper is generally well-written, and I was able to follow most empirical conclusions.

Weaknesses

- The paper only performs experiments on a single model class and task, so it is not clear if these results generalize. For example, the paper observes that most models perform best after training on only 25% of the data, but it is not obvious if this is an artifact of the specific models used or the data. Does the same observation hold true on a different model or a different task? - The performance of DPO seems under-tuned compared to the results of DPO in the literature. (1, 2). - The paper does not provide any details regarding how model selection was performed and additional design choices. (1) REBEL: Reinforcement Learning via Regressing Relative Rewards by Zhaolin Gao (2) BPO: Supercharging Online Preference Learning by Adhering to the Proximity of Behavior LLM

Questions

- Could you provide best-fit lines for each model size in Figure 4 so that I can see trends? (This is the point raised on lines 196-200.) - Is it a fair conclusion that IPO performs the best among the DAA studied in this paper? - Are IPO and SCLI also prone to length exploitation? If so, could you provide results similar to those for DPO? - How did you perform model selection?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes