Summary
This paper proposes the REBEL algorithm that reduces policy optimization to iteratively solving squared loss regression problems on the difference in rewards between trajectories, based on DPO's analysis. The paper transforms the resulting equation for r(x, y) presented in DPO to a regression loss function, and avoids the intractable calculation of Z(x) by calculating the loss based on a pair of samples from the same input prompt x, i.e., (x, y) and (x, y'). One of the goals for REBEL is to serve as a simple and lightweight RL algorithm that eliminates the need for complex components like value functions and clipping heuristics used in PPO. The authors provide a theoretical analysis showing that Natural Policy Gradient can be seen as a special case of REBEL under some assumptions. The authors conduct two kinds of empirical analysis including language modeling and image generation tasks to demonstrate the performance of REBEL.
Strengths
- Originality:
- This paper presents a new angle by transforming the analysis of the reward function presented in the DPO paper into a reward regression loss, leading to the proposed REBEL algorithm.
- The authors make connections between REBEL and existing RL methods like NPG considering some assumptions, showing that these algorithms can be seen as special cases or approximations of REBEL under certain conditions.
- Quality:
- The paper provides a thorough theoretical analysis comparing REBEL with existing RL approaches.
- Clarity:
- The paper is well-written and easy to understand, with a clear logical flow from motivation to theoretical analysis to empirical validation. The authors do an good job of explaining the intuition behind REBEL and highlighting its connections to prior work.
- Significance:
- The paper tackles the important problem of developing simpler and more efficient RL algorithms that can scale to large-scale generative model fine-tuning.
Weaknesses
1. Insufficient experimental validation and limited baseline comparisons:
- While the paper presents empirical results on language modeling and image generation tasks, the experimental validation of REBEL could be more comprehensive. The authors should consider including a wider range of benchmarks and datasets to demonstrate the generality and robustness of their approach.
- The comparison with baseline algorithms like PPO and DPO is somewhat limited. The authors should provide more details on the hyperparameter settings and training procedures for the baselines to ensure a fair comparison. Moreover, the poor performance of DPO compared to PPO in the experiments raises questions about the implementation or hyperparameter choices.
- The authors claim that REBEL matches the strongest known theoretical guarantees in terms of convergence and sample complexity. However, the experiments only compare performance at a specific epoch without demonstrating improved sample efficiency. Convergence plots showing the performance of REBEL and baselines over the course of training would provide a clearer picture of the sample efficiency and convergence properties.
2. Lack of support for certain claims and limited exploration of key aspects:
- The paper makes several claims regarding the advantages of REBEL, such as its ability to handle intransitive preferences, incorporate offline datasets, and apply to deterministic MDPs. However, there is a lack of corresponding experimental evidence or theoretical analysis to substantiate these claims.
- The relationship between the regressor's performance and the quality of the dataset used for training is not explored in depth. Insights or experiments that investigate how dataset quality and diversity affect the regressor's ability to capture an improved policy would strengthen the paper.
- The choice of base distribution \mu is mentioned as a determining factor for whether REBEL is hybrid or fully online. However, the paper does not provide experimental results comparing different forms of \mu across various tasks or practical guidelines for choosing \mu in real-world applications.
3. Inconsistencies and potential conflicts with previous statements:
- The authors mention that critic-based variance reduction might be necessary for high-variance trajectory-level rewards in stochastic MDPs, which seems to contradict the criticism of PPO's complexity in the introductory section. The lack of experimental support for REBEL's performance in stochastic MDPs is a significant limitation, and the authors should provide preliminary results or theoretical insights to support their claims.
Questions
1. Sample efficiency and convergence guarantees:
- The authors claim that REBEL matches the strongest known theoretical guarantees in terms of convergence and sample complexity. However, the experiments only compare performance at a specific epoch without demonstrating improved sample efficiency. Can the authors provide experimental results that support their claim of improved sample efficiency compared to other algorithms?
- It would be helpful to see convergence plots that show the performance of REBEL and baseline algorithms over the course of training, rather than just at a selected epoch. This would provide a clearer picture of the sample efficiency and convergence properties of REBEL.
2. Relationship between regressor performance and dataset quality:
- The authors state that a regressor that can predict the difference in rewards between trajectories implicitly captures an improved policy. Is the performance of this regressor dependent on the quality of the dataset used for training? How does the quality of the dataset affect the regressor's ability to capture an improved policy?
- Can the authors provide insights or experiments that explore the relationship between dataset quality and the effectiveness of REBEL?
3. Applicability to deterministic MDPs:
- The authors mention that REBEL can be applied to any deterministic MDP where the initial state is x and the trajectory y consists of a sequence of actions. Is there any experimental or theoretical support for this claim?
- It would strengthen the paper if the authors could provide empirical results or theoretical analysis that demonstrates the effectiveness of REBEL in deterministic MDPs beyond the bandit formulation.
4. Choice of base distribution \mu:
- The authors state that the choice of base distribution \mu determines whether REBEL is hybrid or fully online. Can they provide experimental results that compare different forms of \mu across various types of tasks? What are the practical guidelines for choosing \mu in real-world applications?
- Insights into the impact of different choices of \mu on the performance and behavior of REBEL would be valuable for practitioners looking to apply this algorithm.
5. Stochastic MDPs and the need for critic-based variance reduction:
- The authors leave the experimental validation of REBEL in stochastic MDPs for future work but mention that trajectory-level rewards can be high-variance, potentially requiring critic-based variance reduction. In what practical situations would the transition dynamics be stochastic? If critic-based variance reduction is needed, how does this align with the introductory section's criticism of PPO's complexity?
- The lack of experimental support for REBEL's performance in stochastic MDPs is a significant limitation. Can the authors provide any preliminary results or theoretical insights that support their claims about REBEL's applicability to stochastic environments?
6. Performance comparison with baselines:
- In the experiments conducted by the authors, DPO performs significantly worse than PPO, especially in Table 1, where DPO is inferior in every case. Can the authors provide an explanation for this discrepancy? Is it due to differences in implementation or hyperparameter settings?
- In Figure 3, the comparison between PPO and REBEL is made at an intermediate checkpoint where REBEL observes a higher reward under the reward model. Is it possible that PPO has already overfit at this selected epoch? How was this specific epoch number chosen for REBEL? What would the comparison look like if the best-performing epoch for each algorithm were considered? Additionally, why is the comparison limited to only PPO? It would be informative to include other state-of-the-art RL algorithms in the comparison to better understand the relative performance of REBEL.