Reward Collapse in Aligning Large Language Models

Response to Reviewer vqeA

Thank you for your valuable feedback! We respond to your comments as follows: > The paper claims that we should use prompt-based utility function Eq.(3). However, all follow analysis is about the utility function without the prompt, which is inconsistent. Our theoretical analysis focuses on determining the reward distribution when a fixed utility function is chosen. Therefore, for the purposes of our analysis, we can fix a utility function. > The paper does not provide the implementation details of the prompt-aware training. In our experiment in Section 3.2, we randomly assigned each question as either open-ended or concrete. For open-ended questions, the utility function is selected as G(x) = -1/x. For concrete questions, their utility function is chosen as G(x) = x. This choice of utility functions serves the specific purpose of demonstrating that the prompt-aware approach can effectively prevent reward collapse. Additionally, in Appendix A.2, we delve into potential practical methods for assigning prompt types.

Response to Reviewer hn6B

Thank you for your support of our paper. We address the questions as follows. 1. The assumption that the LLM is sufficiently overparameterized is quite strong, but it’s essential for our conclusions. Since if the reward does not maximize the objective function exactly, reward collapse may not exists. For example, in the experiment in Section 3 (Figure 1 and 4), reward collapse does not show up in the early stage of training. This overparameterization assumption is satisfied in most real applications. For example, InstructGPT se a 6B reward model [1]. 2. While we acknowledge the potential improvement of RLHF in current LLMs, the primary focus of this paper is to document and investigate the phenomenon of reward collapse during reward model training in LLMs. We present theoretical results on reward collapse and derive the reward distribution under a specific class of utility functions. Although intuitively our approach should enhance the performance of RLHF, we believe that the impact of using a prompt-aware utility function requires thorough further research. Numerous techniques implicitly avoid the harm of reward collapse, such as early-stop or adding regularizers, which may alleviate reward collapse. Investigating the extent to which the trained reward model enhances the capabilities of large language models, such as their ability to self-calibrate uncertainty, is an interesting direction for future work beyond the scope of this paper. 3. Aassigning prompt types is fundamental in the prompt-aware approach, as discussed in Appendix A.2. A straightforward method for determining the prompt type is to gather input from human labelers, who typically rank different responses, as seen in InstructGPT. Additionally, we can request them to assess how open-ended the prompt is using a scale ranging from -1 to 1. Automated annotation processes are also possible. For instance, one approach involves assessing the variability of responses to a given prompt. If the responses exhibit high variability, the prompt can be classified as open-ended. Conversely, if the responses show low variability, the prompt may be deemed close-ended. Determining the prompt type is indeed a complex and fascinating task, providing a promising avenue for future research. [1] Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744. We sincerely value your feedback, and we hope these clarifications address your concerns. If you have any further questions or suggestions, please feel free to let us know.

Response to Reviewer C1CM

Thank you for taking the time to provide feedback on our work. We appreciate your thoughtful comments and would like to address each point accordingly. 1. In Section 3.2, we specified that each question was randomly assigned as either open-ended or concrete. For open-ended questions, their utility function is chosen as G(x) = -1/x. For concrete questions, their utility function is chosen as G(x) = x. This is only for the purpose of our experiment. Additionally, in Appendix A.2, we delve into potential practical methods for assigning prompt types. 2. We understand your concern about the validation of our findings on real-world datasets. While Section 3.1 presented experiments on a real-world dataset to demonstrate the existence of reward collapse, we used synthetic datasets in Section 3.2 to illustrate how prompt-aware training can mitigate this issue. We acknowledge the potential for stronger validation with real-world ranking datasets and agree that it is an area for improvement. Nevertheless, our controlled experiments, combined with those in Section 3.1, effectively convey the key message: reward collapse can be avoided through prompt-aware training. 3. The primary focus of our paper is to document and investigate the phenomenon of reward collapse in the training of reward models for LLMs. We have developed theoretical results on reward collapse and derived the reward distribution under a specific class of utility functions. While we intuitively believe that our approach enhances the performance of RLHF, we recognize the need for further research on the effects of using prompt-aware utility functions. We also acknowledge the existence of other techniques that implicitly address reward collapse, such as early-stop or regularizers. Exploring how the trained reward model enhances the capabilities of large language models, including their ability to self-calibrate uncertainty, is an intriguing avenue for future research, albeit beyond the scope of this paper. 4. Yes, the dashed region signifies the randomness across 128 distinct prompts. We appreciate your suggestion, and in response, we will employ an alternative utility function to ensure a clearer presentation of the results. 5. The distinction lies in the utilization of two different parameters, denoted as $\theta$ (Appendix D.2), in the BTL model. The y-axis represents the value of the empirical cumulative distribution function (ecdf) of rewards obtained from solving the optimization problem: $$ \max_{0\le r_1,\cdots, r_n \le 1} S(r_1,\cdots,r_n), \mbox{ where }S(r_1,\cdots,r_n) = \sum_{1\le i,j\le n} U(r_i - r_j) \sigma(\theta_i - \theta_j). $$ We will ensure that these details are appropriately highlighted for better clarity in our presentation. We sincerely value your feedback, and we hope these clarifications address your concerns. If you have any further questions or suggestions, please feel free to let us know.