Robust Reinforcement Learning from Corrupted Human Feedback

Summary

The paper presents a robust reward learning approach by formulating it as an $\ell_1$-regularized maximum likelihood estimation problem. And it also introduces an alternating optimization algorithm, which introduces minimal computational overhead when compared to the standard RLHF approach.

Strengths

* The paper is well-written and contains two types of experiments: robotic control and text generation. * The method is straightforward, and the motivation is reasonable.

Weaknesses

* The paper lacks an experiment for formal RLHF, such as Proximal Policy Optimization (PPO), in the context of text generation. Including such an experiment could provide a more comprehensive evaluation of the proposed method. * The use of $\delta$ as a regularization method may appear somewhat trivial for this approach. Conducting additional experiments on HH-RLHF or Ultrafeedback to provide further evidence and support for the effectiveness of this method would be beneficial.

Questions

* It is unclear how far the Perturb Percentages can be extended before the model completely collapses. Further investigation and experimentation are needed to determine the limits and thresholds that lead to model failure. * The difficulty level of optimizing $\delta" is not specified. It would be helpful to elaborate on the challenges associated with optimizing this parameter and any potential limitations encountered during the optimization process. * In order to gain a comprehensive understanding of the effects and limitations of the proposed method, I hope the author can conduct experiments specifically focused on PPO.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The context lacks information regarding the limitations of the paper or the proposed method. Besides, the analysis on DPO does not fully reflect the true impact of RM's signal on PPO, such as overrated scores. Hence, to adequately assess the effect of the real RLHF's RM on PPO, it is essential to conduct experiments specifically targeting PPO. The current approach lacks experimentation on PPO, which limits our understanding of how the RM's signal truly impacts this particular method

Summary

The paper proposes a robust RLHF method which models the potentially corrupted preference label as sparse outliers. They prove that their method can consistently identify outliers in addition to learning the underlying reward functions, under proper conditions. The results on both robotic control and natural language generation tasks show that the proposed approach improves robustness of reward learning phase within RLHF framework.

Strengths

This article raises an important issue in RLHF, that human annotators may give incorrect or inconsistent preference labels. and then introduces a simple but useful extension to the existing RLHF framework — including a perturbation factor. The authors theoretically prove the statistical rate of convergence in fitting the reward functions as well as the outliers, under particular conditions. They conduct thorough experiments over different noise modes of human preferences in robotic control, and further extend their approach to DPO within real-world language land to support their claims.

Weaknesses

- Error bars are expected in Table 1. - The Anthropic Helpful and Harmless dialogue preferences dataset has a high disagreement rate, so that flipping a random portion of the training labels may not increase noise as expected, which could probably cause the unexpected results. The authors should probably filter the dataset or try a cleaner dataset to show the results on perturbation.

Questions

- What’s the relationship and difference between Win Rate and Winning Score in your Table 1? - Could you try to draw a fair comparison between Data Filtering and $R^3M$-DPO? Although your Data Filtering baseline doubles the training cost, it still indicates that filtering the dataset could be as effective as modeling the outliers. One thing you could provide is the performance of $R^3M$-DPO over the filtered dataset.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper does not discuss the limitation of the work. Please discuss the potential issue of applying your approach to real-world RLHF tasks.

Summary

The paper studies the problem of robust reinforcement learning when a small fraction of the human feedback preference data can be corrupted by adversary.

Strengths

The paper formulates the robust RLHF problem and provides a straightforward and easy-to-use $\ell_1$ regularization algorithm for learning the reward signal. It provides experimental justification for the proposed algorithms.

Weaknesses

I have a few questions regarding the formulation and experimental results. Please see next section.

Questions

1. In practical dataset, how do we justify the correctness of sparse perturbation assumption rather than stochastic noise in the original Bradley-Terry model? 2. What is the criterion for tuning the parameter $\lambda$ in the regularized loss? 3. The TL;DR and HH dataset might not lead to significant improvement over preferences due to both responses being too much worse than current llama2 generation. Is it possible to run on modern preference dataset like HelpSteer2, Nectar or UltraFeedback, and report the Arena-Hard score as the evaluation?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Summary

This paper proposes a framework for robustifing learning from human preferences. It models noise and bias in human annotation of the dataset as an offset added to the true margin between the preferred and disprefered examples. It further utilizes L1 regularization to induce sparsity in the offset. Empirically, the method is benchmarked on robotics and natural language through its extension to Direct Preference Optimization (DPO).

Strengths

- The paper is generally well-written - The method is well-motivated - Empirical results are generally comprehensive, across two domains, and a hand-full of baselines are considered.

Weaknesses

- The value of the $\delta$ offset could also be interpreted as by how much is the positive completion preferred over the rejected. So in a way, the method is modeling an explicit different in quality of the preference pairs. However, this offset may be orthogonal to label-noise e.g. examples that have small offset (positive and negative are close to one another) may be more difficult to annotate as there is not striking difference between the two completions. However, the statement on line 113 > ... the annotator is very likely to give an incorrect preference only takes into account the opposite extreme. Prior to this work, "Direct Preference Optimization with an Offset" explores DPO with an offset to model the difference in degree of preference (ODPO). - Given that the canonical framework for RLHF of in LLMs is using REINFORCE or PPO with a learned reward model (through logistic regression) , results in that setting are necessary given the framing / positioning of the paper, where the application to DPO is a plesent bonus. In that setting, the test-set classification accuracy of the reward model would of interest in the the different noise settings.

Questions

- I am a bit concerned about the results in in Figure 3, from reading off the graph, the increase / decrease in Winning score when changing the perturb % is about the same for $R^3M$ and DPO for dialogue? It would be clearer if the authors simply plot the delta in the score as that's the metric underlying raw performance, that we care about in this context. - More ablations along the extremes would help ground the experimental results further and increase the quality of the work. e.g. for $\lambda$, higher perturb % (100% perturbation would give a sense of how big the drops in performance are relative to their practical worst-case) -Reporting the ranking accuracy when training DPO (i.e. % of time $r_w - r_l > 0$) with and without the robustification scheme would further strengthen the results.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

No limitations have been provided.

Thanks

Thank you for your response. I appreciate the author’s efforts in running new experiments and most of my concerns are resolved. I have increased my score accordingly. One tiny suggestion is that llama2 might be too weak for arena hard. It might be better to start with at least llama 3 / 3.1 / Gemma 2. But I understand that during the short time window of rebuttal it can be very hard to finish the experiments. I’m excited to see the results in the future version of the paper!

Any Further Questions

Dear Reviewer hEuH, Thank you again for the detailed and insightful review. As the discussion period is ending, we are wondering if you have any further questions or comments we can address. If our rebuttal has addressed your concerns, we would sincerely appreciate it if you would consider raising your score.