Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment

Reply to Reviewer RwZ8

Thank you very much for reviewing our manuscript and providing valuable feedback. Below is a response to the review questions/comments. We have revised the manuscript accordingly as shown in red. Please let us know if further clarifications are needed. **Weaknesses:** The earlier work [1, 2] of this line in standard RL needs also to be mentioned in the paper for the completeness of literature review. **A:** Thanks for your suggestion. We have added [1,2] (respectively as examples of pessimistic offline RL and optimistic online RL) to the end of the "Overoptimization" part in the introduction, as shown in red. **Q1:** In Theorem 1, the choice of the parameter $\eta$ depends on the quantity of partial coverage coefficient $\mathcal{G} _ {\mathcal{D}}$, which somehow seems impractical. How does the learner know the exact number of the partial coverage coefficient? Even though in practice the parameter is typically chosen via empirical search, but this theoretical choice still does not make sense to me. **A:** Thanks for your suggestion. In the revised Theorem 1, we have removed $\mathcal{G} _ {\mathcal{D}}$ from the stepsize $\eta$ and accordingly changed the error bound. **Q2:** Beyond the experiments in the paper, how to understand does the proposed algorithm indeed handle the challenge of corrupted data, reward over-optimization, and verbosity? In other words, is there additional experiment which supports that the superior performance of the new algorithm is exactly brought by the mitigation of these issues? For example for reward overoptimization, can the authors show that compared with the algorithm that does not consider this explicitly, the "reward" is indeed not over-optimized. Similar questions for corruption and verbosity. **A:** Thank you for your suggestion. We are working on these additional experiments. **Q3:** What about the capabilities of the LLMs trained by DPO-COV in complex reasoning tasks, e.g., math and coding? **A:**: Thank you for your suggestions. We compare our DPO-COV algorithm with other DPO variants on Grade School Math 8K (GSM8K) and AI2 Reasoning Challenge (ARC) tasks by following [1] and adapting the code from [2]. The accuracy results below indicate that our DPO-COV algorithm outperforms the other variants also on the complex reasoning tasks. We are going to add these experiments to the revised paper. [1] Liu, Z., et. al. (2024). Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. ArXiv:2405.16436. [2] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://github.com/huggingface/alignment-handbook, 2023. **------------------------------------------------------------------------------------------------------------------------------** **Model** $\quad\quad\quad\quad\quad\quad\quad\quad$ **GSM8K** $\quad\quad\quad\quad$**ARC (Easy)** $\quad\quad\quad\quad$ **ARC (Challenge)** **------------------------------------------------------------------------------------------------------------------------------** **Our DPO-COV** $\quad\quad\quad\quad\quad$ 46.78 $\quad\quad\quad\quad\quad$ 72.52 $\quad\quad\quad\quad\quad\quad\quad\quad$ 49.32 Robust DPO $\quad\quad\quad\quad\quad\quad$ 46.25 $\quad\quad\quad\quad\quad$ 72.14 $\quad\quad\quad\quad\quad\quad\quad\quad$ 47.35 Pessimistic DPO $\quad\quad\quad\quad~$ 45.19 $\quad\quad\quad\quad\quad$ 72.14 $\quad\quad\quad\quad\quad\quad\quad\quad $ 46.16 Length-regularized DPO $\quad$ 44.50 $\quad\quad\quad\quad\quad$ 72.31 $\quad\quad\quad\quad\quad\quad\quad\quad$ 46.16 Vanilla DPO $\quad\quad\quad\quad\quad\quad$ 45.26 $\quad\quad\quad\quad\quad$ 71.89 $\quad\quad\quad\quad\quad\quad\quad \quad \~ $ 46.50 Reference Model $\quad\quad\quad\quad$ 42.38 $\quad\quad\quad\quad\quad$ 71.72 $\quad\quad\quad\quad\quad\quad\quad\quad$ 45.14 **------------------------------------------------------------------------------------------------------------------------------** **Q4:** Typos: inconsistent notation for the partial coverage coefficient, see Section 3.3. **A:** Thanks for pointing that out. In the revised paper, we have changed all $\mathcal{G} _ {\mathcal{D}}$ to $G _ {\mathcal{D}}$.

Authors' reponse to Reviewer 25JV

Thank you very much for reviewing our manuscript and providing valuable feedback. Below is a response to the review questions/comments. We have revised the manuscript accordingly as shown in red. Please let us know if further clarifications are needed. **Q1:** There is no obvious novelty in the algorithm design and theorems. **A:** This combined algorithm for the first time can solve corruption, overoptimization and verbosity issues simultaneously within one simple implementation, which is important in practice. **Q2:** In Theorem 1, it should be $G_{\mathcal{D}}$ rather than $\mathcal{G}_{\mathcal{D}}$? **A:** Thanks for pointing that out. In the revised paper, we have changed all $\mathcal{G} _ {\mathcal{D}}$ to $G _ {\mathcal{D}}$.

Please engage in a non-dismissive way

Dear Reviewer, Please engage with the authors in a non-dismissive way. They spent time writing a rebuttal. Can you please go through their arguments and respond to them instead of just saying "yes". Thanks!

Which paper do you think uses our techniques for analyzing $\xi^{\pi}$ and $\xi^*$?

Thank you very much for reevaluating our work and raising your rating. You think our techniques for analyzing the noise terms $\xi^{\pi}$ and $\xi^*$ have been proposed in the literature. Could you tell us which paper use the same techniques? To our knowledge, such noise terms $\xi^{\pi}$ and $\xi^*$ in RLHF/DPO algorithm design are only used in [1]. Their theoretical analysis and result are different from ours. Specifically, as to the analysis technique, right below Eq. (A.1) of [1], they apply second-order Taylor expansion to $\mathcal{L}(r,\xi^{\pi})-\mathcal{L}(r^*,\xi^*)$ (translated to our notations) followed by strong convexity of $\mathcal{L}$ to obtain linear and quadratic terms of the reward estimation error $\Delta r=r^*-r$ and noise estimation error $\Delta \xi=\xi^{\pi}-\xi^*$. They compare $\xi^{\pi}$ with $\xi^*$ directly. In contrast, we compare $\xi^{\pi}$, $\xi^*$ respectively with zero noise $\xi=0$ and did not use strong convexity of $\mathcal{L}$. As to the theoretical result, [1] only obtains upper bound for reward estimation error, while we obtain upper bound for optimization error of the policy we finally aim to learn.

Reply to Reviewer c44H

Thank you very much for reviewing our manuscript and providing valuable feedback. Below is a response to the review questions/comments. We have revised the manuscript accordingly as shown in red. Please let us know if further clarifications are needed. **Q:** How fine should we set the grids and how sensitive are the results to the hyperparameters? **A:** Great question. We first perform a coarse-grained search: $\eta\in${$0.00005, 0.0005, 0.005, 0.05, 0.5$}, $\omega\in${$0.00005, 0.0005, 0.005, 0.05, 0.5$}, $\lambda\in${$0.05, 0.1,0.2,0.5, 0.7$}. Then we perform a fine-grained search around the best results obtained from the coarse-grained search. The algorithm is sensitive to hyperparameters in the coarse-grained search but is quite stable in the fine-grained search stage. In general, we should choose small $\eta$ and $\omega$ at the order of $[1e-4,1e-3]$, and $\lambda$ smaller than but close to 1.

Authors' Response to Reviewer mVWJ

Thank you very much for reviewing our manuscript and providing valuable feedback. Below is a response to the review questions/comments. We have revised the manuscript accordingly as shown in red. Please let us know if further clarifications are needed. **Q:** Using Llama-3-8B-Instruct it seems the current win-rate can be 20\%+ (see Table 1 of [1]]). The authors reported ~7\% win-rate of using Llama-3-8B. Could you explain the reason here? [1] https://arxiv.org/pdf/2405.00675 **A:** Great question. The results are not directly comparable for two reasons. First, we apply Lora to compress $\theta_{t+1}-\theta_t$ (the change of the big model) to save time while [1] did not. Second, Table 1 of [1] and our experiments for online algorithms use different models for labeling and initial policy $\pi_0$. Specifically, we use Llama-3-8B only to generate the labels $y_t$ and the zephyr-7b-gemma-sft-v0.1 model to initialize the policy $\pi_0$. In contrast, Table 1 of [1] uses GPT-4 to generate the labels $y_t$ and Llama-3-8B-Instruct to initialize the policy $\pi_0$.

Authors' 3rd Reply to Reviewer 25JV

Thanks for bringing the question. By further investigation, we found the following additional novelties. In the algorithm design, we need to notice that the length-penalized reward $r_{\omega}(x,a)=r(x,a)-\omega|a|$ is applied to only the relative value function $V_{\beta,\omega}(\pi,r)$ not the log-likelihood function $\mathcal{L} _ {N,\lambda}(r,\xi)$. In the theoretical analysis of the offline algorithm (i.e., Theorem 1), the true and estimated noise terms are analyzed not only in different stages as mentioned in the paper, but also in different ways. Specifically, in (b) of Eq. (74) at the end of the proof of Theorem 1, the first term $\min _ {\xi\in\mathbb{R}^N}\mathcal{L} _ {N,\lambda}(r,\xi)$ is replaced by its equivalent $\mathcal{L} _ {N,\lambda}(r,\xi_r)$ for the estimated noise $\xi _ r \in {\arg\min} _ {\xi\in\mathbb{R}^N}\mathcal{L} _ {N,\lambda}(r,\xi)$, while the second $\min_{\xi\in\mathbb{R}^N}\mathcal{L} _ {N,\lambda}(r,\xi)$ is upper bounded by $\min _ {\xi\in\mathbb{R}^N}\mathcal{L} _ {N,\lambda}(r,\xi^*)$ for the true noise $\xi^*$.