The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

Summary

The paper discusses more than 20 implementation details of RLHF with PPO using the tl;dr summarization task. The authors achieve great performance on the summarization task, and the authors reproduced the scaling behavior by OpenAI by experimenting with models of different sizes. So, the implementation details must be meaningful.

Rating

7

Confidence

4

Ethics flag

2

Reasons to accept

In general I believe this discussion is a great contribution to the community, as top-performing models’ RLHF procedures are sometimes hidden and not published. The authors also promise to open-source the code. The appendix is helpful (with implementation details and detailed results and prompts). Some of the details are great: for example, “including the leading space in completion” (detail 3) is a common bug in implementation and people rarely openly discuss it, so I really appreciate the authors for writing it out. Another example is detail 7. It is also great because people don’t usually discuss it often. Detail 23 (eos trick; if there’s no eos in the trajectory then assign it a negative reward) is also something I used before, but I didn’t hear others discussing it. So it’s great to see some confirmation that others are using it too and it works.

Reasons to reject

(Detail 19) On DPO vs. PPO reward modeling ability: it’s unclear to me whether the comparison is fair. For DPO reward model, do you mean using the probabilities as an implicit reward model? In that case, would it be more fair if you not only include your regular reward model validation accuracies, but also PPO-trained LM’s reward modeling accuracy (in other words, similar to DPO, treat the PPO-trained model as an implicit reward model, and compute the validation accuracies)? Imo the advantage implementation is quite important but there seems to be minimal discussion. There have been quite a few recent “implementation details of PPO” online. I thought it would be best if the authors highlight the least known and the most important details, and also point out which detail likely doesn’t matter much. I’m not entirely sure if the example in Table 1 is appropriate so I leave it to other reviewers / the meta-reviewer. I know we are all adults but I also don’t want to run into this genre unprompted, especially in an academic paper (e.g., losing my virginity, leaving me with a boner). -- Updated score after rebuttal

Questions to authors

Detail 23 seems important but it’d be great if the authors can elaborate what the values are. Are values for each batch? The past k steps?

Ethics concerns details

Mildly sexual content in Table 1 but fixable

Summary

This paper summarizes the reproduction details of the OpenAI work: RLHF for summarization. The authors study many different items that are specific and important for reproduction. The authors have found several key points that affect the performance.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

1. This is a good manuscript for others to reproduce the work of RLHF for summarization. 2. The tricks are detailed in lots of ways and aspects, which are important for reproduction. 3. The experience could be possible to transfer to other applications.

Reasons to reject

1. Though the comprehensive study appears, it is somehow hard to show the connections between these different details, this potentially could be crucial. 2. The application summarization task is specific, however, it would be better to make some details that could be transferable. Or it should be study some of these items. 3. The base model GPT-2 is somehow old now, at least LLaMA would be a better replacement.

Summary

This paper contributes a detailed reproduction of OpenAI’s RLHF work in the TL;DR summarization. The nice thing about the work is that many technical details (datasets with tokenizer, truncation, padding, EOS tokenizing, disabling dropout, the way they extract reward from the EOS token, etc) are transparent. This should make the paper a good reference for future effort to replicate RLHF work in the dataset. The work also demonstrated the scaling behavior of PPO across different model sizes, this is unsurprising though.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

* This paper will be a good reference for future work on replicating RLHF work in the TL;DR summarization.

Reasons to reject

* The novelty of the work is limited. I consider it as a "white" paper that to unblock everyone who tries to replicate "RLHF" work, which is to me a significant contribution. But other than that I don't see other novelty of the work.

Summary

The paper does a comprehensive reproducibility experiment on the RLHF results from a past OpenAI paper. The authors experiment with different details of training that are often overlooked or not discussed in detail in most papers.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

I think this paper is a great read and a guide to all practitioners in the community working in the intersection of LLM and RLHF. It would

Reasons to reject

I am not sure how much it might align with the "research novelty" aspect of the conference. I leave that judgement to the editors. I would like to point out though that the authors mentioned that many of the practices are not standard in the community. In that note, it would be good to contrast the design choices of the authors against the out-of-the-box choices presented in many open source libraries like deepspeed-chat, trl, etc. Since the main objective of this paper is to make the engineering practices more reproducible and "open", it is important to contrast the presented choices against other available alternatives.

Summary

The paper proposes a re-implementation of the RLHF algorithm with PPO. The paper provides insights into the key implementation details for the reproduction of OpenAI's seminal TL;DR summarization work. Each key implementation insight has been backed by an ablation study into its effect on the final performance. This thorough examination enhances our understanding of the algorithm's intricacies and will serve as a valuable resource for the research community.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

The paper's strength lies in its careful study of the different implementation details in RLHF. The paper demonstrates the scalability of RLHF with various model sizes and validates the effectiveness of the RLHF approach for the pythia models at different scales. Furthermore, the authors have made the complete source code, model checkpoints, and training metrics publicly available. I believe, future works can build on these models for a deeper dive into the model's behavior with RLHF. Furthermore, the paper conducts consistency and calibration studies on the trained reward models. The authors showcase the consistency of these models with human preferences, as judged by external metrics like GPT-3.5, and also report comparisons of their trained reward model to dpo's reward model. They point to the specific differences between the two algorithms. These strengths highlight the paper's contributions to advancing the understanding and application of RLHF, particularly in the context of summarization tasks.

Reasons to reject

I don't have many concerns with the paper as-is. Here are a few questions about the experiments to the authors. a) (Detail 2) How do the experiment observations change with different restrictions to the length of the query tokens (say 256 v/s 512 v/s 768)? b) (Detail 4) How do the differences in SFT and Preference datasets affect the final performance of the trained model? Is there a control study, where the authors restrict the length of the examples in both the datasets to check their impact on the final performance? c) (Detail 21) What's the effect of multiple epoch training on the same SFT prompts? Do the authors observe differences in the quality of the different SFT prompts? d) "Reward whitening makes the model generate shorter outputs." Is this helpful only for summarization tasks?

Questions to authors

Please check my questions above.

Thanks for the rebuttal

Thanks the authors for the rebuttal. I look forward to your next versions of these details. It's okay for the model side. Therefore, I will raise the score.

[Area Chair Comment] Author response

Dear reviewers, Please take a look at the author's rebuttals and the other reviews for this paper! If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum. Please feel free to comment on reviews (or their author responses) other than your own! Thanks!