Summary
The paper addresses a weakness of DPO: it can not deal with preference data whose number of responses is larger than 2. To extend DPO, the author proposes Noise Contrastive Estimation (NCE)-based Alignment algorithms InfoNCA. The author has shown that DPO is one of the special cases of InfoNCA. To further fix the Decreased Response Likelihood issue, the author also proposes NCA with a similar derivation. Experiments demonstrate that NCA and InfoNCA outperform baselines with reward datasets for training, with NCA showing significant performance in complex reasoning tasks like math and coding. The paper offers the community a general framework for LLM alignment from a novel perspective.
Strengths
1. The paper considers the LLM alignment task from a very novel perspective of Noise Contrastive Estimation. The proposed InfoNCA and NCA methods are not only general in form (DPO can be regarded as one of the special cases), but also elegant in math (the connection of LLM alignment and InfoNCE is quite interesting). The paper brings a brand-new view to the study of LLM alignment, which can be insightful to the research community.
2. The motivation of the paper is clear, which focuses on the limitation of DPO when training with multi-response preference data. This improving direction is quite valuable for both the research and industry domains.
3. The experimental results support the effectiveness of InfoNCA and NCA, which outperform baseline methods such as DPO, IPO, and KTO.
4. The paper is clearly written, well organized, and easy to follow.
Weaknesses
1. Fundamental Assumption: My major concern is about the fundamental assumption when deriving the InfoNCA and NCA methods. The author assumes that one of the scored responses is sampled from the optimal policy, which is practically impossible because most of the preference data are sampled from non-optimal LLM policy. In some extreme situations, all the generated responses can be harmful, while our ideal optimal policy should output harmless responses, leading to a contradiction with the author's essential assumption. A possible solution to this weakness might be mixing the human written (golden) response into the preference data, then applying NCE to the prompt and the golden response, combining InfoNCA with some prior work such as APO [1] and SPIN [2].
2. Lack of Human Evaluation: Although the author has conducted GPT-4 evaluation in the experiments, the real human evaluation results are not reported, which should have made the paper's claims more persuasive.
3. Lack an experiments of two-response alignment: Although the paper demonstrates better performance using multi-response preference data, it is interesting to compare InfoNCA, NCA, and DPO on a binary-response preference set.
4. Lack of baseline comparison: Although DPO cannot deal with multi-response preference, one can first train an RM with the multi-response preference, and next sample two responses with scores from the learned RM. Then DPO can be conducted on the RM-scored pairwise preference data. This can be a simple but important baseline that the author should take into consideration.
References:
[1] Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game, ACL 2024
[2] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024
Limitations
The author has NOT addressed the limitations of the proposed method. One of the most important limitations can be that if all the scored responses are harmful, the proposed methods can never achieve the ideal alignment optimum, which could even enhance the harmful model outputs.