Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

Summary

The commonly used preference alignment approach involves training a reward model and optimizing the output of the LLM against this reward model. The effectiveness of the reward model directly impacts the preference alignment. Recently, some studies have attempted to improve reward models through the use of a reward model ensemble technique, which can help avoid the phenomenon of reward hacking. This paper further explores the use of the reward model ensemble in preference alignment and presents some insightful conclusions based on experiments in the summarization and dialogue tasks.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. This paper is easy to read, and the conclusions are clear. This paper explores RM ensemble through reinforcement learning and re-ranking, and evaluates the effects of the data distribution and reward model type. 2. Building high-quality reward signals for preference alignment training is crucial. This paper can help in understanding the reward behavior and better constructing the reward ensemble, thus outputting higher-quality reward signals for the preference alignment.

Reasons to reject

1. As shown in Table 1, although T5-XXL can outperform the best T5-XL model on each task, using the win-rate computed by T5-XXL as an evaluation metric for each task is inadequate. One concern is that T5-XXL exhibits a significant amount of reward hacking behavior, similar to that of the previously used reward models, such as T5-Large and T5-XL. Therefore, it is recommended to also use other metrics (e.g., BARTScore and GPT-4 Win Rate) to further evaluate the policy model. 2. Given OpenAI's reward scaling law, one might wonder if a larger reward model could effectively mitigate reward hacking. It raises a natural question: Is it more beneficial to use a larger reward model, or consider an ensemble of smaller reward models? Furthermore, what would be the impact of introducing more larger reward model into the reward model collection for ensembling? However, these questions are not discussed in this paper. 3. The reward model ensemble discussed in this paper overlooks a significant issue: the differences in score margins between the reward models intended for ensembling. These differences can lead to error results for the reward model ensemble. For instance, consider two samples, a and b. The first reward model assigns scores of -5 and -6, respectively, while the second reward model assigns scores of 3 and 20, respectively. When applying the reward ensemble for re-ranking, the negative score assigned by the first reward model to sample b can be easily overlooked. However, we cannot guarantee that the margin measure is consistent across all reward models. There is a possibility that the first reward model deems sample b to be significantly inferior to sample a, despite only a one-point difference in their scores.

Questions to authors

1. This seems to be a typing error in the sentence: “PALM-2 on the two possible orderings....” 2. Have you tried using list-wise data to train the reward models? If so, is this phenomenon the same as with pair-wise data? 3. Has the reward score of each reward model been normalized or undergone other preprocessing steps?

Summary

This paper investigates the problem of reward hacking in language model alignment using reward models. The authors show that reward models trained on the same data but with different random seeds can disagree significantly on out-of-distribution data generated by the policy during the alignment process. This underspecification of reward models propagates to the aligned policy, making it highly tuned to the specific reward model used during training. To mitigate this issue, the authors propose using reward model ensembles, particularly "pretrain ensembles" where each member is pretrained with a different random seed. Experiments on several language tasks demonstrate that pretrain ensembles are more robust than individual reward models or "finetune ensembles" where members only differ in the finetuning seed. However, the authors find that even pretrain ensembles do not eliminate reward hacking, as they can fail to capture uncertainty and penalize certain undesirable behaviors that are incentivized by all ensemble members.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The paper analyzes the problem of underspecification in reward modeling, providing empirical evidence of its prevalence and downstream effects on the aligned policy. - The proposed solution of pretrain ensembles is well-motivated and evaluated on multiple tasks with different alignment strategies (best-of-n and RLHF). The results convincingly demonstrate the advantages of pretrain diversity. - The authors identify important failure modes of reward ensembles through qualitative analysis, showing that they don't fully solve reward hacking. This provides a balanced and insightful perspective on the limitations of the proposed approach.

Reasons to reject

- Pretraining multiple reward models is computationally expensive, especially at larger model scales. The paper doesn't discuss the trade-off between ensemble diversity and training cost. - I think RMs might not be significantly different from each other, even though they were trained with different seeds. To promote much more diversity among the RMs, a better strategy for the RMs’ training would be explored, e.g., training each RM with a fraction of preference datasets like K-fold cross-validation or ensembles of RMs trained with different alignment datasets. - They did not compare the ensemble of small RMs and a single larger RM in terms of reward overoptimization. In some cases, training a single large RM might be more affordable.

Summary

This paper studies the use of reward models in aligning language model with human preference with a focus on addressing the reward hacking using reward model ensemble. In RLHF, the policy may exploit the model error to achieve a good performance measured by the imperfect reward model, which is usually called reward hacking or reward over optimization. The authors investigate how well reward model ensemble can mitigate this issue during training and inference. Further, the author also study why does reward model ensemble will fail sometimes. The key findings in this paper are: 1. Reward models are underspecified, which leads to different rewards or rankings when tested on out-of-distribution data. 2. Overoptimization occurs when alignment to one reward model does not improve reward as measured by another. 3. Reward ensembles, especially pretrained ensembles (with different random seeds), help mitigate overoptimization and improve generalization. 4. Reward hacking is not completely eliminated even with reward model ensemble. This is because the reward model ensemble will underestimate the uncertainty if it's far away from the data distribution.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

This paper investigates a critical issue: aligning language models with human preferences. It provides a thorough examination of how reward model ensembles can mitigate over-optimization. The finding that ensembles composed of members pre-trained using different seeds generalize better offers valuable insights for practitioners, albeit unsurprising. Overall, the paper is well-organized, and the experimental results solidly support each argument, making it an enjoyable read.

Reasons to reject

The new insights from the paper may seem limited after reading it. The use of an ensemble of uncertainty quantification methods to mitigate reward hacking is not novel in the literature, as many reinforcement learning papers, especially those on offline RL, adopt a similar approach to address the reward hacking problem. The authors hypothesize that the same phenomenon will occur in other approaches for uncertainty estimation that are not aware of distance. I wonder if the authors have tried using SNGP on top of the reward model to make it distance-aware when quantifying uncertainty. If the hypothesis holds true, it should perform better than an ensemble in mitigating reward hacking. Additionally, to validate the above conjecture, it would be interesting to see how increasing the number of ensembles affects performance. If the issue is largely due to being "distance unaware," increasing the number of ensembles should not be helpful beyond a certain point. Implementing this may require significant resources, and it might be impractical to obtain several large pretrained models with different seeds.

Summary

This paper looks at the issue of reward hacking with reward model ensembles. The datasets, models, and training methods are all from previous, except for how different individual reward models are trained and ensembled together. Experiments first show that different reward models predict similar scores for in-distribution data, but disagree substantially for out-of-distribution data. Experiments with reward ensembles show that ensembles usually provide higher performance in best-of-n reranking as well as RLHF finetuning. However, reward hacking is still observed even when reward model ensembles are used in RLHF. Strengths: - The issue of reward hacking and the prevention of it are important problems. - The experiments presented in the paper are thorough and support most of the claims in the paper. - Analysis and discussion provide good insights into the issue. Weaknesses: - Lack of human evaluation. The only human evaluation is qualitative in nature. - Ensembling is a well known method to reduce overreliance of spurrious features. The paper does not contribute to the method itself.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

See strengths.

Reasons to reject

See weaknesses.