Aligning language models (LMs) with preferences is an important problem in natural language generation. A key challenge is that preferences are typically provided at the sequence level while LM training and generation both occur at the token level. There is, therefore, a granularity mismatch between the preference and the LM training losses, which may complicate the learning problem. In this paper, we address this issue by developing an alternate training process, where we iterate between grounding the sequence-level preference into token-level training guidance, and improving the LM with the learned guidance. For guidance learning, we design a framework that extends the pairwise-preference learning in imitation learning to both variable-length LM generation and the utilization of the preference among multiple generations. For LM training, based on the amount of supervised data, we present two minimalist learning objectives that utilize the learned guidance. In experiments, our method performs competitively on two distinct representative LM tasks -- discrete-prompt generation and text summarization.
Paper
Similar papers
Peer review
Summary
This paper aims to tackle the misalignment between sequence-level preferences and token-level language model in NLG. The authors design an iterative training framework that integrates the sequence-level preference into token-level training guidance, mitigating the granularity mismatch. Experiments are conducted on two different LM tasks - discrete-prompt generation and text summarization, indicating its effectiveness.
Strengths
1. The methodology of decomposing sentence-level preference into token-level preference presented in the paper is both intuitive and rational, providing a feasible approach to address the granularity mismatch in language model training. 2. Through comparative performance with baseline models, comprehensive ablation studies and discussions, the paper demonstrates the proficiency and potential benefits of the proposed model.
Weaknesses
1. The paper lacks a precise definition of 'preference,' appearing to suggest it is human bias towards superior text quality. However, the method's experimental use of the METEOR score as a sequence-level reward contradicts this assumed understanding. This ambiguity may lead to confusion among readers and raises questions about the evaluation methodology. Why using METEOR as rewards can improve the evaluation in terms of ROUGE scores? Human evaluations should be conducted for a more comprehensive assessment. 2. The reported performance on CNNDM and XSum tasks, measured in ROUGE scores, is below benchmark levels. A basic BART model reportedly outperforms the proposed method (44 vs 40 ROUGE-1 score), which questions the effectiveness of the new training process. This raises concerns that the method may not improve upon existing architectures as suggested. 3. The absence of an accompanying code with the paper is a significant drawback. Without this, it is challenging to reproduce the results, especially given the complexity of applying reinforcement learning. This could limit the wider verification and applicability of the presented method.
Questions
Can the performance be enhanced by integrating the token-level feedback with RL-based baseline models? These baseline models appear to show promising performance as depicted in Figure 2.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
yes
Summary
To fine-tune an LM, the paper proposes “token-level guidance” by leveraging sequence-level preference. The algorithm alternates between two stages: (1) learning token-level “guidance” (aka reward function) and (2) fine-tuning LM using the “guidance”/reward. To aggregate token-level rewards, the authors propose aggregation functions that are different from the classical summation: average, soft maximum, and soft minimum. Figure 1 is a good illustration of the learning algorithm. It’s similar to RLHF, except the preference is based on multiple generated sequences, the reward is token-level, and the aggregation function can be different. Two “minimalist” LM training objectives are proposed: When there’s no supervised data, the authors use a REINFORCE-style update (with a max-entropy regularizer). When we have enough data, the authors use a reward-weighted MLE objective – the weight depends the importance of the token, which is essentially the normalized token-level reward. Experiments are done on prompt generation (generating discrete prompts so that the accuracy of the corresponding task is good) and text summarization. The performance is competitive with respect to baselines shown in Table 1 and Table 2.
Strengths
It’s great that the authors have done ablation that removes reward retraining, as shown in Section 4.3 (b). Good to see that the performance, after removing retraining, is still competitive with respect to baselines. The token-level reward idea is worth pursuing, and I’m glad to see work in this direction. The experimental results should benefit the community. In addition, I agree with the intuition that summation is not necessarily the only / optimal approach for aggregating different token-level rewards, and it’s great to see that the authors have attempted other aggregation functions.
Weaknesses
Reward function retraining seems expensive. Is there an analysis on the compute cost? I just want to make sure that the authors have ensured that their ROUGE computation is fair. BART paper and the “Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization” paper have different ROUGE scores for lead-3 baseline, which is a bit weird. BART’s lead-3 result is higher. I just want to make sure in Figure 2, for example, the ROUGE scores are all comparable. Related: Is there a reason why Table 2 results are run using T5-small instead of a larger model? Would the trend still hold when T5 gets a lot larger? My understanding is that the reward is Meteor in summarization experiments. Although the summaries are evaluated by ROUGE, what are the Meteor scores (the actual rewards being optimized) on dev/test set? This detail seems to be missing, but apologies if I missed it.
Questions
A few questions are in the "weaknesses" section.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
I don't see significant discussion on potential limitations. Please let me know if I missed those paragraphs.
Summary
The paper proposes to solve the issue of granularity match in preference-based tuning of LMs (RLHF for e.g.,), that is task-based preference is defined at the sequence level (via pairwise preference learning) while the reward model training and policy optimization is done at the token level. The paper proposes an extra step to ground the token-level reward into the sequence-level preference. Evaluation is done on two tasks (prompt generation and summarization) and some improvement is shown compared standard RL-based method such as PPO and NLP as well as vanilla supervised models
Strengths
- The paper is well written and explains the proposed method well and the experimental setup is well documented. - The studied problem is interesting and relevant especially since RLHF methods are becoming more common nowadays. - Evaluation is done against strong baselines. - The proposed method seems to bring some improvements on the two studied tasks
Weaknesses
- **Limited tasks**: Evaluation is only two tasks and it's not clear why these two particular tasks were selected. I imagine other tasks should be used, where preference-based learning is relevant. Such tasks include toxicity avoidance and controllable/constrained generation. - **No human evaluation**: No human evaluation is done on summarization: We know that RL methods are good at hacking metrics and I would expect human evaluation on the summarization task to support the improvements in ROUGE. - **Some choices are not justified**: For example, in Algorithm 2, why is the reward model retrained only for the first half of the iterations? Why not for the entire LM training? Why is a max-entropy gradient added to REINFORCE in Eq 5? Did you add the same for the baselines? - **Not clear where improvement compared to the baselines comes from**: For example in Algorithm 2, the reward model is trained alternatively with the LM. Are all baselines trained the same way? If not, then there is no way to know if improvement comes from this method of training or from the sequence-level grounding.
Questions
- See last two points in Weaknesses - In the ablation in section 4.3, (b), How is this done? you just replace $f(\lbrace r_\phi(s_t^k, a_t^k)\rbrace)$ with $\mathcal{R(s^k_T)}$?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
I think the authors addressed some of the limitations of their work.
Summary
This paper proposed break the pairwise sequence preference into token-level guidance signal by iterating between learning a token-level reward from sequence level preference and improving LM with the learned token level guidance. Experiments are conducted on two language generation tasks and competitive results are reported.
Strengths
- The paper propose to address a problem of how to effectively ground sequence-level preference into LM finetuning. - studies different aggregation function to break sequence level preference into token level reward function - the setup with supervised data to weight token level MLE by token level reward is interesting
Weaknesses
- in general the paper is difficult to follow, e.g. Figure 1 is not self explanatory, experimental setups and datasets could be use more details. - a core motivation/hypothesis of the paper between sequence and token level losses is questionable. E.g. abstract mentions "a granularity mismatch between the preference and the LM training losses". It is wrong to categorize LM loss as token-level loss, because it is MLE over target sequence on both token level and sequence level. there are many unsupported assumptions around the mismatch of sequence and token level losses in the introduction section. - in PPO, even though the reward function is sequence level, there is a token-level value network. Discussion and ablation of how that is trained is very relevant to choosing the aggregation of your token level reward model. - the dataset to test your method is not proper. "simulate the sequence-level preference by the classical Meteor score and report the standard ROUGE scores" is questionable because it is not real sequence level feedback. On the other hand, there are datasets readily available, e.g. "Learning to summarize from human feedback" which contains real human preference data on summarization tasks.
Questions
NA
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
NA
Summary
The paper presents a new approach to training language models that address the mismatch between coarse-grained sequence-level preferences and fine-grained token-level rewards. With more fine-grained rewards, the proposed framework reduces the reliance on supervised data. Specifically, given the preference of a set of sequences, it first trains a reward function such that the aggregated rewards over tokens can satisfy the preference. Then, it applies REINFORCE to update the language model. The framework is evaluated on two text generation tasks: generating text prompts for few-shot text classification and text summarization. Compared with baselines, the proposed method improves the performance on both tasks.
Strengths
Revisiting the mismatch between sequence-level and token-level feedback in fine-tuning language models (LLMs) is of particular interest to me. In scenarios where supervised data for a target domain downstream task is scarce, and fine-tuning LLMs is computationally expensive, exploring data-efficient methods becomes important. The proposed method is straightforward and reasonable to me.
Weaknesses
1. Using discrete-prompt generation to evaluate the proposed method is not convincing. The chosen task itself lacks meaningfulness, and a better way to achieve good performance may be self-training as introduced in [1]. I can understand the goal of the experiment is to evaluate the quality of the generated sequence, but there are other more reasonable generation tasks such as personalized chatbot. 2. The performance of the proposed method of text summarization falls significantly short of the SOTA, which leaves me unconvinced by the results. For instance, the ROUGE-L score for the current SOTA on CNN/DailyMail has surpassed 44, and for XSum it stands at 40.4 [2]. In contrast, the proposed method achieves scores of 38.17 and 26.33 respectively. For CNN/DailyMail, the results are only slightly better than the trivial lead-3 baseline. Considering that the proposed method is complex, I expect more improvement. I can understand the low performance may be partially due to model size. However, without using the model size of a state-of-the-art model, it’s hard to evaluate the effectiveness of the proposed method because It’s easier to boost the performance on a small model. In [3], they challenge the effectiveness of RL and argue that using a few human-curated prompts and responses is more effective for finetuning LLMs. 3. Using ROUGE as the only evaluation metric for summarization is not convincing. At least human evaluation or language model scores such as BERT-score/BART-score should be used. Also, is it possible to obtain preference via language model scores? 4. The proposed method lacks novelty. Previous studies have extensively discussed the mismatch between token-level and sequence-level rewards. Using a token-level reward function to estimate the reward for each token is not a new idea. Several related works are missing [4][5]. [1]ExploitingClozeQuestionsforFewShotTextClassificationandNatural LanguageInference, Schick et. al. [2]BRIO:Bringing Order to Abstractive Summarization, Liu et. al. [3]LIMA: Less Is More for Alignment, Zhou et. al. [4]SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient, Yu et. al, 2016. [5]Unsupervised Text Style Transfer using Language Models as Discriminators, Yang et. al.
Questions
1. Are there any generated samples for the discrete prompts for zero-shot classification? Qualitative analysis may make the method more convincing. 2. How does the choice of reward function impact performance? Does using different pre-trained models as reward functions significantly impact the performance? 3. Why the summarization results are significantly below SOTA results? The paper mentions it uses the full training set for finetuning the model, so I expect the performance should at least not be far from SOTA results. 4. How to obtain the preference for discrete prompts of zero-shot classification?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Please refer to weaknesses and questions.
response to rebuttal
I acknowledge reading the author's rebuttal which aimed to address some of my concerns. Here's my response to the author's rebuttal: > We retrain the reward model only during the first half of the iterations in order to save the computational cost. I wonder how much compute can actually be saved by this. I'd still argue that this is a non-justified design choice. > We kindly note that the max-entropy gradient may not be applicable to other baselines due to their specific nature/designs. In this case, I would expect an ablation showing that your approach still outperforms the baselines without the max-entropy gradient. I understand that achieving a 100% fair comparison with the baselines is hard, but the authors add extra layers of complexity through some design choices, making a fair comparison much harder than it needs to be. > We respectfully refer the reviewer to our ablation study in Section 4.3 (b), where we remove the reward-function retraining scheme. Thanks for the reference. I can now see that your approach does not rely on the reward re-training scheme, which begs the question as to why it is a part of your approach since the improvement introduced as a result is extremely minor (0.08 ROUGE-1 points, for e.g.) Again, this points to some design decisions made by the authors that are not fully justified. Respectively, it seems like the proposed approach relies on so many moving parts that it has become unclear where the contribution actually is or whether these extra layers of complexity are justified. Overall, I thank the authors for their rebuttal. However, nothing in the author's response merits raising my score so far.
Further Response to Reviewer xDED
Thank you so much for the quick response. We respectfully disagree with your comment that our paper has many unjustified design choices. We address your concerns in details below. > **Q1.** How much compute can be saved by only retraining the reward model during the first half of the training process? **A.** In our preliminary study, we observed that “retraining the reward model only during the first half of the LM training process” can save about 25-30% computation, compared to “retraining the model throughout the entire LM-training process.” Intuitively, “retraining only during the first half” can clearly reduce computation compared to “retraining the model throughout the entire LM-training process.” We believe that the computation-saving nature of this design can sufficiently justify itself. Therefore, we respectfully disagree with the reviewer’s argue that this is an unjustified design choice. > **Q2.** Expect an ablation showing that your approach still outperforms the baselines without the max-entropy gradient. **A.** We first kindly remind that as discussed in **Q3** of the previous Rebuttal, our most direct competitor in the prompt task RLPrompt (Line 246-249), has a max-entropy gradient as well. As a recap, we test our method on the task of discrete-prompt generation. Apart from RLPrompt, the other discrete-prompt baselines use more demanding designs and requirements than our method (or RLPrompt), such as human handcraft and/or other forms of human efforts. Specifically, * “Manual Prompt” uses hand-crafted prompts. * “In-Context Demo” requires (human) selecting one training example per class. * “Instructions” requires manually created task descriptions and label definitions. * “GrIPS” requires instructions designed for humans. By contrast, our method does not require any of these demanding human efforts, and while less demanding, can still outperform those methods. We clarify that the max-entropy gradient is part of our LM-training objective Eq. (5), which targets the few/zero shot setting, where we do not assume the availability of supervised data, e.g., the ground-truth prompts. Due to this challenging setting, we believe that it is unfair for us to remove this component and compare with a baseline having max-entropy gradient (RLPrompt) or with baselines using more demanding designs, such as human handcraft and/or other forms of human efforts (e.g., “Manual Prompt”, “In-Context Demo”, “Instructions”, “GrIPS”). Based on the above clarification, we believe that our comparison with the baselines is fair. In addition to the above clarification, we kindly note that RLPrompt also has max-entropy gradient, and in its paper compares with the same set of baselines as our paper. Since RLPrompt’s experimental comparison has been perceived as fair by the community, we believe that our comparison with these baselines is fair as well. > **Q3.** Why is the reward-retraining scheme a part of your approach since the introduced improvement is minor on the summarization task? **A.** As discussed in Line 314-317 of our main paper, the gain of this scheme depends on the zero-shot ability of the initial LMs. Specifically, in the prompt task where the initial LM has little zero-shot ability, reward-function retraining can particularly be helpful to both improve performance and reduce variance. As a numeric example for Figure 4 in our paper, in the “SST-2” dataset of the prompt task, reward-retraining scheme improves the result from $90 \pm 2.9$ to $92.6 \pm 1.7$. Meanwhile, we agree with the reviewer that *on the summarization task*, the reward-retraining scheme indeed may not help the results as much, since the initial LM has some zero-shot ability (Line 317-319). Since our paper wants to build a general framework that is applicable to both of these distinct settings, we apply the reward-retraining scheme to both tasks in our main results. Then, the ablation study in Section 4.3 (b) serves as a further experimental explanation of this scheme. *** Overall, based on this response and the previous Rebuttal, we believe that our method has similar complexity at least with our direct competitors RLPrompt and RL4LMs. Furthermore, we believe that we have sufficiently validated our method’s building blocks through our comprehensive ablation study in Section 4.3 and Appendix A.2, which is acknowledged by Reviewer **9EYo**. We hope this additional clarification can address all your concerns and merit raising your score. Please kindly let us know if you have any remaining concerns.
Response to authors
Thank you for the rebuttal. I went through the details. Here are a few more comments. Q1 (reward retraining): Thanks for the response. It's great that the authors have taken steps to make reward retraining more efficient. I would appreciate it if there's an analysis on - how much compute is spent on reward retraining vs. other training, - and how much compute the authors' techniques are saving, with respect to regular/vanilla reward retraining. Q3: Good to know that the trends hold on T5-base as well. No need for experiments given that the authors say there is a lack of compute, but do the authors think that the trend will generalize to even larger models? Additionally, I realized that there should be discussion on other weighted MLE approaches in NLP, as I don't think it's an extremely common technique. Just a few examples: RAML in https://arxiv.org/abs/1609.00150, weighted MLE in grammatical error correction for MT in https://arxiv.org/abs/1804.05940, weighted MLE in table-to-text generation in https://aclanthology.org/2021.acl-short.11/, GOLD in https://arxiv.org/abs/2009.07839. Raising my score from 5 to 6, given that I'm satisfied with the other parts of the response.
Raising score further
On second thought, I'm raising the score to 7. I think the token-level reward direction and the different token-level reward aggregation methods (other than summation) are especially interesting.
Response to Reviewer gmdH
We deeply appreciate your feedback and are delighted that our paper and rebuttal merit your raising the score. For reward retraining, we will follow your suggestion and rerun our method to carefully time reward retraining v.s. LM training. We will also rerun the direct baselines RLPrompt and RL4LMs to compare our method’s timing with. These results will certainly be added in the next version of our manuscript. Regarding the saved time, in our preliminary study, we observed that our techniques (“retraining the reward model only during the first half of the LM training process”) can save about 25-30% computation, compared to vanilla reward retraining (“retraining the reward model throughout the entire process”). For the trend, yes, we believe that the trend will continue to hold on even larger models, based on our experiment of scaling up from T5-small to T5-base. Finally, we are grateful to the reviewer for the great suggestion on related works. As a heads up, RAML uses sequence-level reward (negative edit/Hamming distance), rather than token-level as in our method. We will certainly add a detailed discussion on the weighted-MLE methods in NLP, especially your four suggested papers, in our revised manuscript.
Response to the Official Comment by Reviewer xDED
Thank you so much for raising the score. We are delighted that our responses are helpful. We will certainly incorporate all your suggestions into our revised manuscript.
Thanks for your response. I've read the rebuttal and I'll keep my score.
Thank you authors for the responses. I increased my scores after reading your responses and other reviewer's comments. Q2: It is the autoregressive factoring of the LM that ties sequence probability to token probability. Viewing LM cross entropy loss as token-level loss is one view of the problem. An equivalent view is that, if we conduct MLE on sequence probability, which is autoregressively factored to the product of token probabilities, seq_prob = \pi_i token_prob_i, we would get the exact same loss function as summation of token level MLE. Using this view, the cross entropy loss can be explained without mention of token-level losses, and also from data perspective, the SFT data we collected are full sequences (one argument could be that we do not collect token/prefixes but always collect full sequences). Equivalently, any signals that are collected on the sequence level are propagated to token-level because of the autoregressive factoring. I acknowledge that sequence level losses and token level losses are often mentioned in literature, however in my opinion, the there is no mathematically difference on sequence-level and token-level losses for autoregressive LMs.
Further response to Reviewer wRKD
Dear Reviewer wRKD, Thank you so much for your response and your raising the scores. We agree with the reviewer that the token probabilities in LM have connection with the sequence probability. But, as per you commented, *”Viewing LM cross entropy loss as token-level loss is one view of the problem”*. Therefore, we believe that our core motivation is solid, and it is valid to categorize LM loss as token-level loss. We agree that there are other views of the same problem, but we do not preclude those other views and we do not aim at drawing a dichotomy. We will add a clarification on this in our revised manuscript. We would like to reiterate that the goal of this paper is to ground sequence-level feedback into (dense) token-level guidance for LM training (Section 1). In our experiments (Section 4), we demonstrate that our viewpoint and method can be beneficial on two distinct representative LM tasks/settings. Specifically, with supervised data, even though the classical supervised MLE can work, we show in Section 4.2 that our token-level reward-weighted MLE can perform better. Intuitively, the reward/guidance-weighting scheme emphasizes the important tokens in the supervised sequences and downweights the unimportant ones, and therefore can better utilize the LM capacity and the optimization budget (Line 167-171). In the challenging setting of without supervised data, the LM needs to discover each token by itself. We show in Section 4.1 that our dense token-level guidance can improve the performance/quality of the generated sequences. In both settings, we experimentally show that our dense token-level guidance can be more effective for LM training than the delayed/ungrounded *native* sequence-level feedback (Line 246-249 and Line 276-278). *** We wish this discussion can clarify the contribution of our paper and merit raising your rating further. Please kindly let us know if you have any remaining concerns.
Decision
Accept (poster)