Response to Reviewer 7o3q (Part 1)
Thanks for your detailed review.
### Regarding Weakness 5
This is a misunderstanding. **The results in Table 1 represent downstream task performance that is calculated by commonly adopted metrics (arc-challenge: acc, arc-easy: acc, gsm8k: exact_match, and truthfulQA: bleu_acc)**. LPC is mainly developed from direct preference optimization, which does not involve an explicit reward model. Besides, we do not involve any additional reward models to rate the generated responses. As a result, **our method does not produce reward scores**.
### Regarding the effectiveness of variational inference and its application in NLP (Weakness 1 & 2)
We acknowledge the concern about the robustness and training stability of LPC. This is a common issue suffered by most latent-variable methods. However, it is important to note that LPC exploits discrete codes (a.k.a., vector quantization) rather than continuous variables (e.g., variables following Gaussian distributions) as latent representations of human preference. This makes a big difference, as vector quantization does not have to perform sampling from distributions, which can effectively circumvent the variance issues, as indicated by [1]. In fact, all results presented in Table 1 and Table 2 are averaged after 5 runs (each run corresponds to a different random seed). The variance of DPO w. LPC ($var_{average}=5.8e-5$) and that of DPO w/o. LPC ($var_{average}=5.1e-5$) are comparable. Detailed variance statistics for the results in Table 1 are given as follows:
| Methods | Arc-challenge | Arc-easy | GSM8K | TruthfulQA | Average |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Llama3-8B DPO w/o. LPC | 1.2e-4 | 1.7e-5 | 4.5e-5 | 5.3e-5 | 5.8e-5 |
| Llama3-8B DPO w. LPC | 4.2e-5 | 4.9e-5 | 4.1e-5 | 7.4e-5 | 5.1e-5 |
| Llama3-8B SimPO w/o. LPC| 8.2e-5 | 4.1e-5 | 1.8e-4 | 3.4e-5 | 8.4e-5 |
| Llama3-8B SimPO w. LPC| 6.9e-5 | 4.4e-5 | 1.3e-4 | 3.3e-5 | 6.9e-5 |
To make our results more convincing, we report the preference accuracy of the 5 runs as follows:
| Methods | run 1 | run 2 | run 3 | run 4 | run 5 | Average |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Llama3-8B DPO w/o. LPC | 69.1 | 69.3 | 69.4 | 69.4 | 69.3 | 69.3 |
| Llama3-8B DPO w. LPC| 70.2 | 70.8 | 70.6 | 71.2 | 70.7 | 70.8 |
| Llama3-8B SimPO w/o. LPC| 69.6 | 69.2 | 69.4 | 68.7 | 69.4 | 69.2 |
| Llama3-8B SimPO w. LPC| 72.3 | 71.4 | 71.7 | 72.7 | 70.4 | 71.8 |
From the detailed results above, we observe stable and consistent improvements. We hope these details (as well as the clarification above for Table 1) can convince you regarding the effectiveness and stability of the proposed method.
We understand that there might be different opinions toward latent-variable models. Such methods have been well-established in computer vision [1][2], and attain a lot of interesting progress in NLP, including some recent achievements [3][4]. Then, can we say the exploration on alignment with latent variables is invalid, just because of the tags in our mind for latent variables based on ``past NLP research”? In that case, we would not see the prosperity of PPO in LLM alignment, because for a long time, the community of NLP think RL does not work for NLP [5]. Echoing your comment, we also think variational inference is suitable, intuitive, and rational for preference learning, yet detailed methodology has not been well developed. This motivates the work. We hope our effort can inspire more discussions and studies about variational inference for preference modeling in the NLP community. We also hope the research community could be more open-minded and inclusive, encouraging diverse approaches and explorations.
[1] Aaron van den Oord et al., Neural Discrete Representation Learning. In NeurIPS 2017.
[2] D.P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In ICLR 2014. (Test of Time Award on ICLR 2024)
[3] Aurko Roy and David Grangier. Unsupervised Paraphrasing without Translation. In ACL 2019
[4] Lin Y. C., et al., ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning. Findings of EMNLP 2024.
[5] Rajkumar Ramamurthy et al., Is Reinforcement Learning (NOT) for natural language processing: benchmarks, baselines, and building blocks for natural language policy optimization. In ICLR 2023