Response to reviewer us9M
Dear Reviewer us9M,
Thank you so much for your positive review and insightful comments. We appreciate your feedback and have incorporated your suggestions into the revised version to improve it.
**#1: Regarding your suggestion on larger and more recent LLMs**
We have conducted experiments using the latest 7B LLaMA model. In our study, we compared SFT, RLHF, and CoH against Koala, a state-of-the-art LLaMA-based chatbot, serving as a robust calibration baseline. Our comparison (see Figure 9) of these methods applied to LLaMA on HH and summarization datasets with Koala used pairwise human evaluation. Notably, Koala is developed from the ShareGPT dataset, known for its superior quality and greater diversity compared to open-source datasets like HH and summarization. The results indicate that our CoH approach performs on par with Koala, and when combined (CoH + Koala), it slightly outperforms Koala based on human ratings. However, both C-SFT (conditional SFT) and SFT significantly lag behind Koala, highlighting the effectiveness of CoH in leveraging human preferences for learning.
**#2: Clarification on "past tokens" and masking ratio**
By “past tokens,” we refer to the causal tokens visible to a given token. Our approach randomly masks past tokens to prevent overfitting during training. The masking ratio was determined through preliminary experiments; we found that a larger ratio hindered training efficiency, while a smaller one made negligible difference.
**#3: On regularization and pretraining data**
For the regularization, we used approximately the same number of tokens as the human feedback data. This approach resulted in about a 40% increase in wall clock time for training. We will clarify this further in our revision.
**#4: Suggestion for GPT-4 based evaluation**
While we primarily focused on human evaluation for its reliability, we acknowledge its limitations in terms of scalability and expense. We have conducted a GPT-4 evaluation for LLaMA-trained SFT, PPO-RLHF, and CoH, following the MT-Bench evaluation framework. Our results show that CoH achieves the highest score, significantly outperforming SFT and PPO. We will include results of GPT-4 based evaluation for more tasks in the revision.
| Model | Turn | Score |
|-------|----------|-------|
| SFT | 1 | 5.34 |
| PPO | 1 | 5.82 |
| CoH | 1 | 6.03 |
| Model | Turn | Score |
| SFT | 2 | 4.23 |
| PPO | 2 | 4.56 |
| CoH | 2 | 4.87 |
| Model | Average | Score |
| SFT | | 4.79 |
| PPO | | 5.19 |
| CoH | | 5.45 |
We appreciate your efforts in using our open-source code. The issue you encountered with model generation quality could be due to the absence of regularization, leading to overfitting, or incorrect hyperparameter settings. We plan to enhance our codebase with more detailed instructions and example scripts to aid future research and ease of use.
**#5: Does inference process need post-processing**
No extra post-processing is required during inference. The model is designed to generate quality responses directly. However, we are exploring potential post-processing methods that might include using search algorithms to select the best outputs or combining various token outputs for exploration. These are promising directions for future research.
**#6: previous submission review feedback and revisions**
In our ICML 2023 submission, reviewers commended the novelty of our work in utilizing pretrained language models aligned with human feedback, as well as the strong empirical results in summarization and dialogue tasks. A primary concern was the absence of an RLHF baseline, which we have now addressed in this submission. Our updated comparison includes both automatic metrics and human evaluation, showing our model's substantial superiority over RLHF.
**#7: Qualitative model inference examples**
We have included examples of model inference outputs in the appendix and plan to add more in the camera-ready version.
Once again, thank you for your valuable feedback.