Summary
The paper identifies four core components in training from preference feedback (RLHF): labeled preference data, learning algorithm, reward model, and unlabeled training prompts; and conducts a study on the effect of each component separately to disentangle their contributions to performance.
The authors experiment with variations of each component and make several claims:
1. There is a clear ordering of the impact of each component on performance: preference data quality > algorithm choice > reward model quality > targeted policy prompts.
2. Synthetic data with (aggregated) per-aspect preferences works best.
3. PPO outperforms DPO
4. Increasing the reward model size or its training dataset size improves its performance on reward benchmarks, but has marginal effects on downstream tasks overall.
5. targeted online training prompts can improve performance in domain-specific settings, but not overall performance.
Hence, they suggest a recipe with the best of each component.
Strengths
The paper is well presented: the claims are clear, the experiments backing them are easy to identify, and the narrative is easy to follow.
The description of the experimental setup is comprehensive and very detailed, helping the applicability of the paper.
The authors experiment with a comprehensive collection of preference datasets and benchmarks.
The authors make several interesting observations from their experiments such as the effects f the preference datasets used showing mostly on dimensions such as truthfulness and instruction following, or the improvement of reward modeling showing in rewards benchmarks but not translating to downstream tasks.
Weaknesses
I find that the claims made in the paper often deviate from the experimental support provided, they are either overstated compared to the scope or the experiments, or inaccurately reflect the scope of the experiments. Specifically,
**Algorithm choice:**
Although comparing components such as prompt datasets is possible when keeping all else the same (algo, hyperparameters, etc), the claim "PPO outperforms DPO" cannot be made without carefully considering hyperparameters.
The authors have a single hyperparameter configuration for each model, and although they may have been validated in other settings, they have not been optimized (within a fixed but same budget) for each algorithm on the author's setting.
The DPO hyperparameters were originally used by Zephyr-Beta, and taken without tuning for TULU 2 [Appendix F.1, 22].
For PPO, apart from the number of epochs that have been cut to 1, it's not clear how the authors obtained the rest of the hyperparameters.
Furthermore, from Table 2, "PPO outperforms DPO" _on average_ on _pre-selected_ datasets. When looking at individual benchmarks DPO models outperform PPO models in quite a number of tasks, and Table 2 only includes a subset of datasets which has not been motivated in the paper.
Therefore I believe that the claim is misleading.
**Preference dataset**:
Only Table 1 with DPO is used to formulate the conclusions on the best preference dataset. For the claim to hold in the scope of the paper (considering its limitations with a single pre-trained model) the preference datasets have to be compared with at least multiple algorithms (and ideally hyperparameters).
Table 2 provides some results for PPO but on all datasets, so it fails to complement Table 1.
The authors do not provide insights about why synthetic data performs best. It is appreciated to observe such difference in performance, but the claim would have more value and impact with some analysis or intuition as to why it would hold.
The claim "Preference-based learning has the strongest effect on instruction following and truthfulness performance." may only be valid in when the preference data itself mainly includes preferences based on instruction following and truthfulness, which is the case in the datasets considered in the paper. However, it may be very valid, but inefficient, to train factuality using preferences.
The scope of such a claim should be carefully explicited.
**Reward model**:
Although the paper aims to disentangle the effects of each component in RLHF, the experiments it conducts on some components tend to depend on the results from the previous components.
E.g. the reward model experiments are only carried out on the preference datasets providing the best performance, giving a biased view on the effects of the reward model and does not allow to formulate an ordering on the performance contribution of each component independently.
Also, the average improvement of 0.6 from the 13B to the 70B UltraF. reward model has been qualified as marginal while the improvement of 0.7 from DPO to PPO has been qualified as significant.