Learning Guarantee of Reward Modeling Using Deep Neural Networks

Response to Reviewer Yeyi (1/2)

Dear Reviewer, Thank you for your constructive comments and for dedicating your valuable time and effort toward the thorough evaluation of our paper. We have carefully considered your comments and the helpful suggestions provided. We have made significant efforts to address your concerns. Please see our point-by-point responses below. **Weakness 1: "about width and depth in design reward models"** Regarding the practicality of the result, we have conducted numerical experiments in our revision with an example for studying how the configuration of network architecture (depth and width) affects the regret of the learned estimators (Please see Page 21 in the Appendix and the attached PDF in Supplementary Material in the rebuttal revision). Our simulation results show that the regret of the learned network estimations can achieve similarly low levels for a large region of values in the choices of the network depth and width as long as they are proper (not too small and not too large). In other words, the regret of the learned network estimations is robust to the choices of network width and depth. This observation is in line with our theory where proper order of network depth and width can lead to optimal rates. It offers a practical guideline for implementing our approach in real applications. Our empirical example also supports the theoretical guide regarding the importance of network depth. As illustrated, increasing the depth demonstrates superior performance gain compared to increasing the width with a similar number of parameters. Our findings on the efficiency of network depth and the robustness of network architecture choices (networks with different architectures achieving comparably good performance) have also been observed in the literature [1]. **Weakness 2: "about writing and presentation"** Thank you for your suggestion. In this revision, we have tried our best to improve the presentation of our manuscript based on your suggestions. (1) We would like to clarify that the noise exponent $\alpha$ describes the data quality, and $c_g$ is a constant that appears in the inequality, of which the exact value is unavailable. Both of them are determined by the data distribution but not tied to a specific parameterization of the preference model. Under this assumption, we derive $c^\prime_g$ for both Example 1 and Example 2 to make them adaptive to the inequality in Lemma 1. We would like to mention that Lemma 1 only depends additionally on the log-concavity of the comparison function $g$ and is thus applicable for various pairwise comparison models. We will include more details to improve its clarity in our revision. (2) We sincerely appreciate the reviewer’s questions. In this revision, we have updated the discussion near lines 187-190 to improve its accuracy. In this paper, we mean that the bounds are tight up to unknown constant factors by using the term "information-theoretically optimal." The "information-theoretically optimal" implies that the rate is optimal and cannot be improved further without additional conditions. We acknowledge that the terminology "information-theoretical optimality" is closely related to information theory [2,3]. (3) Thank you for your helpful suggestion. In this revision, we have made our best efforts to improve the writing, especially in Sections 1 and 2, to make the paper structure more suitable for the conference.

Response to Reviewer 9CkQ

Dear Reviewer, We appreciate your thorough review of our paper and your positive feedback on our work. We have carefully considered your comments and have made significant efforts to address each of your concerns. Please see our point-by-point responses below. **Weakness 1: "about insufficient review on error bounds for dnn"** Thank you for your suggestion and for bringing these important studies ([1,2,3]) to our attention. In this revision, we have acknowledged the valuable theoretical studies referenced above. Specifically, we have included a literature review in Section 4 that focuses on the convergence analysis of deep neural networks, thereby enhancing the comprehensiveness of our discussion. In terms of technical novelties, we emphasized them in the revised version and summarized them as follows: First, unlike typical deep regression learning, our context involves an additional identification problem in reward modeling that needs to be carefully addressed. In the revised version, we emphasize the identification constraint in Lines 276-279 and the corresponding theoretical result in Theorem 3 and 4. Second, beyond the deep neural network structure, our proof considers the influence of the underlying structure of the pairwise comparison dataset. Specifically, we explicitly illustrate how the comparison data impacts the convergence rate via the data coverage assumption. This point is emphasized in Assumption 4 and Theorem 3. To the best of our knowledge, our analysis is the first to explore deep reward modeling under data coverage conditions. Last but not least, our theory reveals the connection between the intuitive notion of clear human beliefs and the reward gap between the optimal action and others. Specifically, we link clear preferences to the probability gap in the dataset rather than directly imposing assumptions on the underlying reward function. This provides a comprehensive explanation of how the margin-type condition contributes to the final convergence rate in reward modeling. Additionally, it offers a theoretical approach to formalizing empirical insights about clear preferences in RLHF modeling. **Weakness 2: "about the optimization error"** We thank the reviewer for providing the insightful work and helping to improve the quality of this paper. We agree that optimization error is a very important but challenging one, which is of independent interest to study [1]. We acknowledge that the stochastic and approximation errors are studied, while the optimization error is not addressed in this paper and we will ensure that this is clearly stated in our revision to provide clarity for the readers. In the revision, we include the relevant discussion based on your provided literature to highlight the importance of optimization error in the theoretical analysis of the DNN estimator. **Question 1: "about the margin condition when α=0"** $\alpha=0$ implies no additional assumption is made on the data distribution, allowing the existence of greater ambiguity in the preference dataset. In this case, the regret bound defaults to the slow rate, and any policy $\pi_{r}$ may require more data to achieve the same level of accuracy compared to situations with α>0. **Question 2: "about unknown smoothness parameter."** Thank you very much for your helpful questions. In practice, the smoothness parameter may not be directly inferred. However, many empirical experiments validate that deep networks yield good performance and can cover the most commonly considered function class. In this revision, we provide a toy example to study the relationship between the performance and network parameters. The details of the experiments are attached in the Appendix. As shown, a relatively flat region corresponds to (nearly) minimum excess risk. It implies that it is unnecessary to calculate the network structure with the smoothness parameter explicitly. Many network structures can achieve satisfactory numerical performance. **Reference** [1] Wang, Mingze, and Chao Ma. "Generalization error bounds for deep neural networks trained by sgd." arXiv preprint arXiv:2206.03299 (2022). [2] Goldenshluger, Alexander, and Assaf Zeevi. "A linear response bandit problem." Stochastic Systems 3.1 (2013): 230-261. [3] Bastani, Hamsa, and Mohsen Bayati. "Online decision making with high-dimensional covariates." Operations Research 68.1 (2020): 276-294.

Response to Reviewer bWGB (1/2)

Dear Reviewer, We appreciate your thorough review of our paper and your positive feedback on our work. We have carefully considered your comments for improvement. Please see our point-by-point responses below. **Weakness 1: "about possibilities to extend to other network structures"** Thank you for sharing your concern regarding the network structure. We acknowledge the gap between these theoretical results and the practical guidance for fine-tuning in reinforcement learning with human feedback (RLHF). The claimed extension mainly considers the reduced complexity of neural networks through the lens of functional equivalence [1]. Such functional equivalence generally exists for fully connected networks, Residual networks, and attention-based networks and it is possible to extend the theoretical convergence analysis for the fully connected networks considered in this paper to other network structures via the functional equivalence approach. [2] derived the convergence rate for transformer networks using a similar analysis based on the stochastic error and the approximation error. And Theorem 5.4 of [2] achieves a polynomial convergence rate, akin to that found in ReLU network analysis. Notably, they also consider the shift-equivariant properties of transformer networks, which supports this claim. In this revision, we have acknowledged and discussed the connection and gap between the fully connected neural networks and attention-based networks and recommend [3,4,5] for more comprehensive theoretical studies of transformers in Lines 391-399. **Weakness 2: "about the applicability of the margin type condition"** We appreciate your insightful comment about the practical applicability of the margin-type condition. Our results provide a precise characterization of RHLF's performance across different scenarios, explaining both sample efficiency and potential limitations up to a margin exponent. Even in cases where the margin condition is not fully satisfied (that is $\alpha = 0$), our analysis shows convergence is still achievable, albeit requiring larger sample sizes and more careful model design. Previous literature ([6,7,8]) has demonstrated valuable practical implementations. For example, [6] mentioned that in the fine-tuning of GPT-3, training labelers agree with each other 72.6% of the time on average despite the complexity of the task, which is quite high. It implies that the “clear human belief” may be more common in practice. [7] proposed using contrastive learning methods to differentiate the data embedding according to preference labels. [8] invokes active learning algorithms to conduct context selection in RLHF. We refer to Section 4 in the main text of our paper for more discussion on recent empirical studies that have developed various strategies for managing crowd-sourced preference data with varying levels of ambiguity. Our theoretical results support these practical implementations, such as enhancing data quality and handling ambiguous samples. As the reviewer suggests, while this approach may not always be the optimal solution, it is a valuable strategy in practice. We recommend the literature in discussions on related work for more details.

Response to Reviewer bWGB (2/2)

**Question 1: "about the network design analysis guiding empirical implementation"** Thank you very much for your helpful questions. In this revision, we have included an example of a simulation study to demonstrate the relationship between network architecture and regret. The experimental details are provided in the Appendix. In particular, our simulation results show that the regret of the learned network estimations can achieve similarly low levels for a large region of values in the choices of the network depth and width as long as they are proper (not too small and not too large). In other words, the regret of the learned network estimations is robust to the choices of network width and depth. This observation is in line with our Theorem 4 where proper order of network depth and width can lead to optimal rates. It offers a practical guideline for implementing our approach in real applications. In addition, our empirical example supports the theoretical guide regarding the importance of network depth. As illustrated in our fixed sample size setting, increasing the depth demonstrates superior performance gain compared to increasing the width, although both yield a similar incremental number of network parameters. Our findings on the efficiency of network depth and the robustness of network architecture choices (networks with different architectures achieving comparably good performance) have also been observed in the literature [9]. **Question 2: "about the extension beyond ReLU networks"** Regarding the extension of Theorem 4 beyond ReLU fully connected networks, we acknowledge the potential for such adaptations, including the attention mechanisms. Recent studies [2,3,4,5] collectively underscore the robust theoretical underpinnings and wide applicability of Transformers and attention mechanisms, supporting the potential extensions and adaptations mentioned. We have included more discussions to mention the potential for future studies in the main text. **Reference** [1] Shen, G. (2024). Complexity of deep neural networks from the perspective of functional equivalence. In International Conference on Machine Learning, 41, 2024. [2] Takakura, S., & Suzuki, T. (2023, July). Approximation and estimation ability of transformers for sequence-to-sequence functions with infinite dimensional input. In International Conference on Machine Learning (pp. 33416-33447). PMLR. [3] Deora, P., Ghaderi, R., Taheri, H., & Thrampoulidis, C. (2023). On the optimization and generalization of multi-head attention. arXiv preprint arXiv:2310.12680. [4] Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., & Kumar, S. (2019). Are transformers universal approximators of sequence-to-sequence functions?. arXiv preprint arXiv:1912.10077. [5] Fang, Z., Ouyang, Y., Zhou, D. X., & Cheng, G. (2022). Attention enables zero approximation error. arXiv preprint arXiv:2202.12166. [6] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730-27744. [7] Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., ... & Jiang, Y. G. (2024). Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080. [8] Das, N., Chakraborty, S., Pacchiano, A., & Chowdhury, S. R. (2024). Active preference optimization for sample efficient RLHF. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models. [9] Han, J., Hu, M., & Shen, G. (2023). Deep neural newsvendor. arXiv preprint arXiv:2309.13830.

Response to Reviewer HieT (1/2)

Dear Reviewer, Thank you for your overall positive comments and for dedicating your valuable time and effort toward the thorough evaluation of our paper. We have carefully considered your comments and made significant efforts to address each of your concerns. Please see our point-by-point responses below. **Weakness 1: "about new insights provided in this paper"** Thank you for your comments. We would like to highlight several key aspects that distinguish ours from conventional ones. We have emphasized these differences in the revised version and summarized them as follows: First, we model clear human beliefs through the selection probability gap between optimal actions and others. Unlike many existing studies, we link clear preferences to the probability gap in the dataset rather than directly imposing assumptions on the underlying reward function. To the best of our knowledge, we are the first to model clear human beliefs using a margin-type condition in Reinforcement Learning from Human Feedback (RLHF). Second, our result incorporates the influence of the underlying structure of the pairwise comparison dataset. Based on the proposed data coverage assumption, we explicitly demonstrate how the comparison data affects the convergence rate. This point is emphasized in Assumption 4 and Theorem 3. To the best of our knowledge, our analysis is the first to address deep reward modeling in the context of data coverage conditions. Third, unlike typical deep regression learning, we carefully addressed the additional identification problem in reward modeling in the context of the comparison model and RLHF. In the revised version, we emphasize the identification constraint in Line 276-278 and the corresponding theoretical result in Theorem 3 and 4. **Weakness 2: "about verification of this margin assumption"** Thank you for sharing your concern regarding the margin-type condition. In this paper, we focus on developing a theoretical understanding of clear human beliefs based on the margin-type condition as many existing studies (listed as follows) emphasize the importance of clear preferences and suggest various approaches for handling ambiguous samples in practice. Our work introduces the margin-type condition as a way to model clear human preferences in Reinforcement Learning from Human Feedback (RLHF) to mitigate the gap in its theoretical understanding. To the best of our knowledge, this is the first theoretical quantification of high-quality data in RLHF. Here we list some existing studies that implicitly mention the importance of this assumption in real-world datasets. [1] mentioned that in the fine-tuning of GPT-3, training labelers agree with each other 72.6% of the time on average despite the complexity of the task, which is quite high. [2] mentioned that the presence of incorrect and ambiguous preferences in the dataset due to the low agreement among annotators during preference labeling could lead to difficulties in accurately representing human preferences. They suggest that detecting and mitigating noisy data is essential for aligning learned rewards with true human preferences. Recently, [3] has given empirical results comparing the performance by training the model with (i) only the clear preference samples and (ii) both clear preference samples and tied pairs. They conclude that including tied pairs in direct preference optimization is not good for task performance. A possible solution includes modeling the ties explicitly ([3]), correcting the labels of wrong preferences, and smoothing the labels of ambiguous preferences to avoid the model’s overfitting on these low-quality data, as suggested by [4].

Response to Reviewer HieT (2/2)

**Question: "Does example 1 or 2 satisfy assumption 1"** Thanks for your questions. We would like to clarify that the noise exponent $\alpha$ is not associated with any specific pairwise comparison probability model. Instead, this assumption pertains to the state (data) distribution and is intended to describe the quality of data obtained from pairwise comparisons. We have acknowledged this point in the main text and provided details in Example 4 and 5 in the Appendix to make it clearer. **Reference** [1] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730-27744. [2] Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., ... & Jiang, Y. G. (2024). Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080. [3] Chen, J., Yang, G., Lin, W., Mei, J., & Byrne, B. (2024). On Extending Direct Preference Optimization to Accommodate Ties. arXiv preprint arXiv:2409.17431. [4] Zheng, R., Xi, Z., Liu, Q., Lai, W., Gui, T., Zhang, Q., ... & Ge, W. (2023, July). Characterizing the impacts of instances on robustness. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 2314-2332).

Response to Reviewer d1ZR (1/2)

Dear Reviewer, Thank you for your overall positive assessment of our paper and for dedicating your valuable time and effort toward the thorough review. We have carefully considered your comments and have made significant efforts to address each of your concerns. Please see our point-by-point responses below. **Weakness 1: "about the assumption on the existence of an underlying reward model and the data coverage assumption"** Thanks for the comments about our assumptions on the reward model and data coverage. We would like to make the clarification as follows. First, the assumption on the existence of a reward model comes from the general reinforcement learning problems [1], especially reinforcement learning with human feedback (RLHF) [2,3,4,5] and inverse reinforcement learning [6]. Since our work studies the learning guarantees of reward modeling in the RLHF, it is unavoidable that we should adopt this assumption. It is still underexplored to model human preference without an underlying golden reward model. We anticipate if there is another systematic approach to explain human preference and be willing to study its theoretical performance in the future. Second, the data coverage assumption is a necessary assumption to guarantee the converging of the maximum likelihood estimator (MLE), which is the most widely used approach in RLHF[3,7]. Specifically, [7] and [8] show that the MLE will fail if the dataset is insufficiently covered. In this work, we further explore how the coverage of the data influences the convergence rate of the MLE. Our result in Theorem 3 reveals the error bound will not be controlled once the 2nd smallest eigenvalue is sufficiently small. How to deal with insufficient coverage issues and build the corresponding learning guarantee are very interesting but challenging problems that deserve future studies. **Weakness 2: "about the novelty of the proofs"** We understand your concern about the novelty of our proofs given the general framework of error decomposition we have used for establishing theoretical results in deep learning. Nonetheless, we would like to highlight three key aspects that differentiate our proof from the standard approach. These have been emphasized in the revised version and are summarized as follows: First, in addition to the deep neural network structure, our proof incorporates the influence of the underlying structure of the pairwise comparison dataset. Based on the proposed data coverage assumption, we explicitly demonstrate how the comparison data affects the convergence rate. This point is emphasized in Assumption 4 and Theorem 3. To the best of our knowledge, our analysis is the first to address deep reward modeling in the context of data coverage conditions. Second, our theory reveals the connection between the intuitive notion of clear human beliefs and the reward gap between the optimal action and others. Specifically, we link clear preferences to the probability gap in the dataset rather than directly imposing assumptions on the underlying reward function. This provides a comprehensive explanation of how the margin-type condition contributes to the final convergence rate in reward modeling. Additionally, it offers a theoretical approach to formalizing empirical insights about clear preferences in RLHF modeling. Lastly, unlike typical deep regression learning, our situation involves the additional identification problem in reward modeling, which requires careful addressing. In the revised version, we emphasize the identification constraint in Lines 276-279 and the corresponding theoretical result in Lines 1073-1076.