Rebuttal by Authors (1/2)
Thank you for your review and feedback. We address your concerns and questions as follows.
For your main issue, we first clarify:
> The theory presented essentially states, "we can partially train an RLHF model, then predict the result of fully training. The resulting predicted model will achieve the accuracy that would have been obtained by full training".
>
> (We first clarify that our term "full training" refers to the baseline that uses the complete training steps. This does not imply that the model is trained to optimality.)
Our formalization in Section 2 does not imply that ExPO is "predicting full training". ExPO hypothesizes that: for a partially-trained model $M_1$, there may exist a better-performing model $M_2$ on the extrapolation trajectory from $M_0$ to $M_1$. This $M_2$ **may not (and very likely cannot)** be obtained by continuing to train $M_1$, given the inherent uncertainty and non-uniqueness of neural network optimization and convergence (for large language models, this also involves learning rate dynamics, gradient clipping, etc.). **ExPO does not aim to, nor can it "predict" the results of full training.**
> ... the results demonstrate that the extrapolation method results in strong gains even when M1 is a fully trained model. (This seems at first like a sign that M1 is undertrained, ...)
As you noted, we conjecture this is because $M_1$ has not been trained to optimality. Given the inherent challenges in training LLMs, alignment training is very likely to be suboptimal (as also evidenced in Section 4.1).
> ... Based on the theory, the extrapolated model M2 should be most accurate if the search for M2 occurs along the line between M0 and the "fully trained" M1.
We respectfully disagree. A fully-trained model (i.e., the baseline using complete training steps) is not necessarily trained to optimality, thus its indicated $\Delta\theta$ may not be optimal (e.g., it may overfit or capture spurious features in the data, as stated in lines 240-242). A partially-trained model may indicate a more accurate $\Delta\theta$, in which case the optimal model on this extrapolation trajectory would outperform the optimal model on the trajectory indicated by the fully-trained model (according to our discussion in lines 266-268, since $\theta_2=\theta_0+(1+\alpha)\Delta\theta$, the achievable performance of $M_2$ is uniquely determined by $\Delta\theta$).
> This line is approximated by training a proxy (e.g. $M_1^{20\\%}$) and searching along the line between M0 and the proxy.
Based on the previous responses, ExPO is not "predicting full training", thus the "proxy" is not approximating the trajectory from $M_0$ to the fully-trained $M_1$.
> But for some reason, this proxy yields *better* results than searching the line between M0 and M1 (22.7% win rate compared to 18.0% win rate). Thus, a misalignment of the search space is beneficial to the model. This seems like a clear sign that something not predicted by the theory is causing the benefits in accuracy.
Based on the previous responses, ExPO is not "predicting full training", and the "proxy" does not necessarily (and likely will not) indicate the same extrapolation direction $\Delta\theta$ as the fully-trained model. Therefore, the experimental results do not contradict Section 2. On the other hand, the performance improvements come from the accuracy of $\Delta\theta$, which is exactly what we discuss and analyze in detail in Section 3.3-3.5.
> Given the above observations, it seems like the true gains in accuracy could be coming from the fact that ExPO models are selected using a method that finds the best-performing model on the UltraFeedback dev set as calculated by another reward model.
We respectfully disagree. As stated in lines 266-268, since $\theta_2=\theta_0+(1+\alpha)\Delta\theta$, and $M_0$ is fixed, the achievable performance of $M_2$ is uniquely determined by $\Delta\theta$. The development set and reward model only assist in searching for $\alpha$, they do not change $\Delta\theta$, and thus naturally cannot change the achievable performance of $M_2$ (although due to limitations of the reward model, the searched optimal $\alpha$ may vary slightly; however, improving reward models to enhance search results is beyond the scope of our work).
> My point here is, it seems like the theory presented doesn't match the given results, and there's some other reason for the gains in accuracy.
Based on the above discussion, we respectfully suggest that you may have misunderstood the idea of ExPO (which is not "predicting full training"), and we have provided clarification and explanation. We hope this addresses your concerns about ExPO's soundness.