Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Summary

This paper proposes *FINE-GRAINED RLHF*: an extension to (the now popular) *reinforcement learning from human feedback* (RLHF) that allows incorporating more granular human feedback into the language model (LM) training process. Typically, RLHF leverages reward/preference models that are trained to output a *scalar* reward for the whole generated sequence, and this *reward sparsity* can hinder improvements for long-sequence generation. Furthermore, scalar rewards do not offer insights into which parts of the sequence were problematic or displayed undesired behaviors. To address this, the using *fine-grained reward models*: for each "category" of feedback of interest, a separate reward model is trained. Furthermore, these models generate feedback at lower granularity (for example, at sentence-level). The policy LM can then be optimized with respect *weighted* sum of the rewards at each timestep, using standard RL algorithms like PPO. The authors explore the impact on two language generation tasks: (1) detoxification and (2) long-form question answering. For detoxification, the authors find that (with a reward based on an external API) training with rewards provided for every sentence yields better performance when compared to a holistic sequence-level reward. However this evaluation is fully automatic. For long-form question answering (QA), the authors introduce a novel dataset, QA-Feedback, accompanied with fine-grained human feedback on initial policy LM. The feedback comes in three error categories at different levels of granularity: (a) irrelevance, repetition or incoherence (sub-sentence level), (b) incorrect or unverifiable facts (sentence level), and (c) incomplete information (whole sequence level). They collect this feedback through crowd-workers. They also collect *preference-based* feedback, for comparison with (hollistic) RLHF. By training separate reward models for each category, the authors show that fine-grained RLHF outperforms the traditional RLHF and supervised baselines, both in automatic and human evaluation. Interestingly, by carefully adjusting the weights of different reward models, the authors show LM behaviors can be fine-tuned to optimize different combinations of desired behavior, and outperform. The authors also explore the reward model accuracy and even compare against a ChatGPT baseline.

Strengths

I really enjoyed reading this paper. It explores a very prominent topic recently (learning from human feedback), it is well written and it does an very extensive analysis (at least for one of the tasks it tackles). “Reward sparsity” is a well known problem in RL, and recent works in RLHF for LLMs just kinda ignores it. Their proposed fine-grained RLHF is seems like a good approach to tackle it, makes intuitive sense and seems easy to implement. Their findings surrounding controllability could also have significant implications for (RLHF’d) LLMs. They also introduce a novel dataset for QA with fine-grained feedback.

Weaknesses

The main weakness of this paper was that only a single LLM was tried: while I believe that their findings will generalize to other LLMs, an experimental validation of this would make this paper even stronger. Even trying re-using the reward models trained for the initial GPT-2 policy on some other LLaMA would already be quite a bonus. Also a couple of other minor ones: - I think task 1 (detoxification) in general is not a good setup: in uses closed source APIs for model-based (from a relatively weak one) PPL as evaluation. the simulated “sentence-level” reward is also not very elegant. I would ideally suggest trying to replace it with another simple, controllable generation task and preferably using human-eval. But ye most of these are covered by task 2 - The multi-reward aspect of the algorithm makes RLHF now a sort of *multi-task* optimization problem, and bring with the problem of how to weight the different categories. The discussion on the trade-off in (4.5) is interesting and helpful, but I think more could be said on how to pick a weighting for a general case where we just want a single “quality”. Also a very simple pareto analysis (probably trying more weightings of the reward) and some plots showing the final trade-off (ideally in terms of human reward?) would add more technical insight

Questions

- The levels of granularity in the paper make sense for the three categories considered, but have you thought about the more general problem of allowing arbitrary granularities? It seems it is mostly a UX/feedback collection process, and that nothing in the algorithm would prevent it. - Is there plans to open-source the (fine-grained) RLHF code? Integrating this into popular packages for RLHF (like TRLX) could heavily boost the impact of this work

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above

Summary

This paper studies how fine-grained RLHF improve over holistic preference RLHF. The study is conducted on two tasks and fine-trained reward comes from sub-sentence, sentence, and full sequence level annotations on different feedback types (toxicity, relevancy, factuality and completeness). Study show that fine-grained reward improved over holistic preference and demonstrated LM behaviors can be customized using combinations of feedback types.

Strengths

- The paper is clearly written with good visual aid, has sufficient details on datasets, reward model training, descriptions on experiment hyperparameters and frameworks. - The paper demonstrates benefits of fine-grained reward model on two tasks, with various reward density and reward types. It is convincing to see it works across very different problem settings. - There is clear and concise analysis on reward model and human rating agreement to understand the quality of reward model. - Experiment sections contains relevant ablations that supports main results and analysis of how different types of reward model works together.

Weaknesses

- (minor) in the detoxification analysis, I'm not fully convinced that the gain comes from reward, it is possible that the API has better accuracy annotating shorter sequence. Some additional analysis similar to your reward model human agreement analysis on task2 could help explain it better. - I didn't find the mention of reward model size which I assume would be the same as the value model of t5-base which is smaller than the policy model. In previous works there is advantages of using larger than policy network value/reward function. It might be important to show what is the effect of reward/value model size on fine-grained reward vs holistic reward. - (major) on task2, there lacks analysis comparing F.G.RLHF vs Pref.RLHF on holistic preference eval. It would be a lot more convincing if there is gain in that eval. Otherwise it could be argued that these different approaches are optimizing toward very different signals. - the comparison ChatGPT using ROUGE score: it is a reference-based score that might not correlated well with human preference. Using ROUGE score itself to compare these systems are not sufficient or even detrimental. Your eval with rel/fact/comp and the holistic preference are much better metrics.

Questions

See weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

NA

Summary

This work improves the reinforcement learning from human feedback (RLHF) with more fine-grained reward signals. For the feedback density, the authors suggest to reward the policy model at sub-sequence level. The experiments on the detoxification task shows that the fine-grained rewards can outperform the holistic sequence level ones. For the feedback diversity, they set three reward categories: 1) **rel.**, irrelevance, repetition, and incoherence, 2) **fact.**, incorrect or unverifiable facts, and 3) **comp.** information completeness. The first two signals, i.e., rel. and fact., are with the sub-sentence and sentence level density, and the comp. is for the full sequence. A new QA-FEEDBACK dataset with fine-grained error annotations and pair-wise preference ranks (for the baselien preference-based RLHF) is constucted for the training and evaluation. Compared with the preference-based RLHF, the suggested fine-grained one behaves better on all three error types in both human and automatic evaluation. This framework also demonstrates the promising ability of customizing LM behaviors by adjusting the weights of different reward models.

Strengths

1. This work suggests a new perspective to RLHF for LM, where fine-grined and denser reward signals can benefit LMs. 2. The proposed fine-grained RLHF framework is simple and effective, validated via thorough experiments and analysis. 3. A new dataset is contributed for the research community, which is carefully created. 4. The paper is well-written and easy to follow.

Weaknesses

1. The models used in this paper are relatively limited in size, only including large and base models. I'm curious how your method would scale to larger models, such as the more commonly used 13B and 30B models, and what kind of performance it would achieve. 2. LM customization is interesting, but lacks human evaluation. I suppose that incorporating human evaluation could help to further understand this.

Questions

1. More different weights settings for the LM customization. 2. Efficiency and speed evaluations of the suggest method and baselines.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

None

Summary

This paper introduces FINE-GRAINED RLHF, a framework that enables training and learning from reward functions that are fine-grained in different error categories. Through experiments on detoxification and long-form question answering, the authors demonstrate how learning with this FINE-GRAINED reward function leads to improved performance, supported by both automatic and human evaluation. The paper's contributions include a new framework for fine-grained reward learning, a new dataset for long-form question answering, and experiments demonstrating the effectiveness of the proposed approach. Though, the proposed method still has some limitations in terms of computational cost, algorithm complexity, and task-binding. In all, it has some contributions to the RLHF field.

Strengths

- The paper introduces a new framework for fine-grained reward learning, which is a novel approach to language model training. The authors also provide a new dataset for long-form question answering, which can be used for future research in this area. - The paper is well-written and clearly presents the proposed framework and experimental results. The authors provide detailed descriptions of their methodology and evaluation metrics, which enhances the quality of the paper. - The proposed framework has the potential to improve the performance of language models by providing more fine-grained feedback during training. - The idea is simple but reveals the potential of RLHF at finer grains.

Weaknesses

- I know that data labeling is cumbersome and expensive, but the 3,853 training data collected in this work is still a bit small, not sure if this is enough to draw the conclusions mentioned in this work. - The proposed method is a bit complicated, it will introduce more computational cost in RLHF, and it will introduce more hyper-parameters for fine-tuning. - The design of T_i^k is not fully explored, which may have a big impact on the performance of the proposed method.

Questions

- The authors say that "the higher the KL coefficient, the lower the perplexity, and the higher toxicity". But only three KL coefficient is tested (i.e., 0.1, 0.2, 0.3) in the experiments, it is not enough to get such a conclusion. Also, what are the results on the generation fluency when the KL coefficient is 0, which means we do not employ a KL penalty? It would be better to explain the KL penalty in more detail. - Why do you choose different models for different tasks? I.e., for the detoxification experiments, the policy model is initialized with GPT2Large, and the value model is initialized with GPT2-base; for the long-form QA experiments, the policy model is initialized with a supervised fine-tuned T5-large, and the value model is initialized with T5-base. - Is the proposed method a good solution in terms of LLM's hallucination?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

- The training dataset is not large enough (especially for the Long-Form QA task), which limits the verification of a conjecture: As the size of training data increases, the effect of the FG-RLHF proposed in this paper will weaken. - The proposed method is expensive, not only in the additional computational cost of the reward model, but also in the different definitions of fine-grained feedback in terms of the feedback types and the density level of each feedback type.

Acknowledgment of Response

I would like to thank the authors for their response. Overall I'm satisfied with responses. I still think the paper would highly benefit from experiments with other models (even if not much larger than the current one), but think this is still a very valuable contribution, and will keep my review score.

acknowledge of response

Thank you authors for the response. I do not have further questions.