The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Summary

This paper focuses on the optimization and learning methods for ``online'' RLHF and contrastive offline methods (DPO, IPO). The authors aim to understand the separation between these two type of methods, where they are different in terms of whether new responses can be sampled or not. The authors state that the difference in the reward parameterization is key to the separation and propose global coverage and local coverage to capture such a difference. The theoretical insights also motivate a novel approach called Hybrid Preference Optimization (HyPO) that combines offline data with online samples, where the online samples are used to control the KL divergence. Empirical results are provided to verify the effectiveness of the proposed methods.

Strengths

- The authors study the theory of RLHF under the KL-regularized target, which is more close to the practice, as compared to many previous works using the non-regularized reward; - The notion of local coverage condition is novel in the literature and is very natural in the analysis of KL-regularized target. I appreciate the paper writing at the beginning of section 4, which is easy to follow and informative. - Building open the notions of local convergence and global convergence, the authors show a clear separation between the offline algorithms like DPO and online RL-based algorithms. This aligns with the recent observations that the online algorithms outperform their offline counterparts with a large margin. - This paper takes a step further to study the differences between the DPO and RLHF. While the original DPO paper states that the learning of DPO is equivalent to the RLHF, the empirical results do not align with this. The discussion related to the parametrization of reward for assumption 4.4 explains such a difference in practice. - Overall, I feel that the story of this paper is complete. Lemma 4.1 and the discussion around assumption 4.4 clearly show that the offline algorithms can search for the policy with a large KL (possibly due to the parameterization of reward). This not only aligns with the separation between global coverage condition and the local coverage condition, but also motivates the practical algorithmic design to explicitly control the KL divergence.

Weaknesses

This is not a weakness but some clarification on the terminology. In the literature of RL theory, particularly for the preference learning paper, the learning (online exploration) is mentioned with querying the human preference so as to learn the $r^*$. In the setup of this paper, the online data is only used to compute the KL loss without querying the human feedback. Therefore, it is more related to an intermediate setting that we use the offline preference dataset to construct a proxy reward function and we are allowed to query the model to get new responses but not the human feedback. See the RSO [1], some discussions in [2] for proposing multi-step RSO, and some discussions in [3]. [1] Statistical rejection sampling improves preference optimization. [2] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint [3] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. Then, it would be interesting to see the empirical results in terms of the ONLINE setting where we can query the new response and the new human preference signals because a lot recent works show that even with the DPO, the online framework outperforms the original one with a large margin. Moreover, this is standard in the PPO literature like Chat-GPT, Claude, and LLaMA2. [4] Training language models to follow instructions with human feedback [5] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback [6] Llama 2: Open Foundation and Fine-Tuned Chat Models

Questions

see weakness part

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

This work considers the statistical separation between contrastive algorithms (DPO and IPO) and RLHF. It proves that DPO/IPO requires global convergence assumption which is in general a very strong assumption while on the other hand RLHF only requires local coverage. This separation stems from the explicit KL constraint in the objective of RLHF while the implicit reward in contrastive algorithms can results in unbounded KL. Given the observation, a hybrid (offline + online) algorithm is proposed: the preference optimization remains offline DPO however it draws online samples to enforce KL constraints on DPO policy, recovering the statistical guarantee of RLHF while attains the practical computational strengths (w/o value and reward models) of DPO.

Strengths

- Well written and easy to follow; understanding DPO and RLHF is important towards better preference optimization algorithms. - Good technical qualities and offer viable insights into DPO. - The proposed hybrid algorithm recovers the statistical guarantee of RLHF while keeps the practical computational strengths of DPO (w/o using value and reward models).

Weaknesses

I only have a couple of minor comments: - Second sentence of L562 should be appended to L231 to make the proof sketch immediately clear. - Theorem E.1 could be relocated to Section 5 to be more self-contained.

Questions

- I might have overlooked, what is the intuition behind online KL evaluation? Is it possible to enforce such KL constraint purely in offline setting?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Overall, I am convinced this is a good submission with good technical quality. However, as I did not extensively follow the theoretical results on preference optimization, I am not able to comment on the novelty of this work, hence I give confidence score of 3. It would also be beneficial if the authors could create a table summarizing previous works and highlighting their contributions in comparison to prior research.

Summary

The paper focuses on the paradigm of fine-tuning large language models (LLMs) using human preference data. It delves into two primary techniques: online reinforcement learning (RL) and offline contrastive methods. The authors challenge the previous notion of these techniques being equivalent by conducting a theoretical analysis through the lens of dataset coverage. They introduce the concepts of global and partial coverage conditions, proving that the former is necessary for offline methods using forward KL like DPO to converge to the optimal policy, while the latter is sufficient for online RL methods using reverse KL. The paper proposes a hybrid preference optimization (HyPO) algorithm, demonstrating its empirical superiority over DPO while maintaining some computational efficiency. The authors also provide a coverage-based explanation for why RL and offline contrastive methods might decrease the probability of preferred responses.

Strengths

1. The paper provides a rigorous mathematical analysis of the different conditions under which offline and online methods have provable performance guarantees, contributing to the theoretical foundation of preference learning in RL. 2. The authors introduce the HyPO algorithm and support their theoretical findings with empirical results, demonstrating the effectiveness of HyPO on the TL;DR dataset.

Weaknesses

1. The experiment is not sufficient. Since it is done with Pythia 1.4B and the TL;DR dataset, it is unclear if the proposed method is still valid on larger models and other datasets. While theoretically interesting, it is unclear if using reverse KL instead of forward KL can lead to a significant performance gain in practice. 2. The proposed HyPO method only uses online samples to calculate the KL loss rather than to collect new preference feedback. While simpler, it may fail to fully leverage the benefits of online methods.

Questions

See the Weaknesses section.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discussed the limitations well in the paper.

Summary

This paper studies learning from human preference feedback for aligning large language models (LLMs). Many existing works share the same observation that offline alignment methods such as DPO underperforms their online counterpart such as PPO. But this phenomenon has not been well understood. This paper studies this difference through the lens of coverage. It unveils that the KL constrain in offline alignment methods can be violated due to incomplete coverage and thus offline methods may find bad solutions. Inspired by this insight, the authors propose a new method, hybrid preference optimization (HyPO), which combines the offline preference update with the online KL regularization. Experiments on the summarization task demonstrate that HyPO is better than DPO. This paper also sheds light on a counterintuitive observation that DPO often decreases the likelihood of preferred responses through an illustrative example.

Strengths

This work addresses an important matter and provides insights on a widely discussed question - why do offline alignment methods often underperform online methods? The paper is well written and easy to follow. The theoretical claims are sensible and easy to understand. I believe the community will benefit a lot from this work.

Weaknesses

The computational cost of the proposed HyPO algorithm is not sufficiently discussed. Usually the most computational cost in online alignment methods comes from the sampling process of LLMs. In practice, the computational cost plays an important role in comparing algorithms. Thus it will provide useful guidance if the computational cost is well discussed in the paper. The discussion in Section 6 is bit handwavy. While Example 6.1 provides good intuition, a more rigorous analysis is needed to justify the claim. One minor suggestion on presentation: It will be good to provide the PPO numbers in Table 1 to help the readers compare HyPO to PPO more easily.

Questions

In the TL;DR experiment, I am curious to know what happens if the authors also train HyPO for 4 epochs. Would it perform better than PPO?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

This paper includes a fairly thorough discussion on the limitations at the end. I would like to add one point: the analysis in this work only focuses on the solution space. It explains why offline methods may find bad solutions, but doesn't explain why they often find the bad ones in practice. As far as I know, people often use a very low learning rate for DPO in practice. For example, the original paper used 1E-6. One possible reason for such low learning rates is that it compensates the ineffective KL regularization. Despite such efforts, DPO still finds bad solutions. It will be insightful if we get a better understanding of why bad solutions seem inevitable.

Thanks for the responses

For DPO-like algorithm, the memory efficiency is not very important because we do not train a value network. The bottleneck of the PPO algorithm is from the challenges of training policy model and value model simultaneously. In particular, the reward model, and reference model to compute the KL divergence can be served with a remote server. Personally, I believe that the coverage condition is hard to satisfy in the context of LLMs and online exploration and online iterative training are standard in making the state-of-the-art models like llama3.1, gemini, Claude, and gpt according to their technical report. In these settings, the reward models are typically trained on a much larger and more diverse preference dataset (compared to the training set of DPO), and is further combined with some human-written rules, and llm-as-a-judge, which are a proxy of the r^* or P^* (see llama 3.1 report or [1] RLHF Workflow: From Reward Modeling to Online RLHF. 2024 for examples). This setting is different from the online setting considered in this paper. I believe that further study in this setting can be promising as future work.

Thank you for the response!

I would like to thank the authors for their response. After reading the authors' response and other reviewers' comments, I believe my initial evaluation is appropriate and thus will maintain my rating.