Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Summary

The paper introduces DPH, a new method for pre-trained language models that addresses the limitations of RLHF. Unlike traditional RLHF, which can compromise a model's reasoning abilities and cause hallucinations, DPH employs an auxiliary reward head to learn human preference signals without altering the LM's output distribution. The authors conduct a theoretical analysis linking their objective function to cDPO and demonstrate that DPH can be used in conjunction with existing alignment techniques to improve performance. The experiment results show that models fine-tuned with DPH achieve higher scores compared to those fine-tuned with SFT or DPO alone.

Strengths

The authors highlight the side effects of RLHF, such as damage to the model's ability to reasoning and hallucinations, and try to solve them. I believe the research problem is significant and worth investigating. The authors are going to release their code and model weights, which would benefit our community.

Weaknesses

1. The claim that the auxiliary reward head avoids affecting the output distribution of the language modeling head is undermined by the practical implementation. Specifically, this claim does not hold when the backbone language model is updated to learn the preference distribution (line 180) and when the model is updated using a joint loss function (Eq. 7). These aspects of the implementation weaken the validity of their claim. 2. Experiment setup: The baseline setup in the experiments is not reasonable. The authors compare their method with other language models that have distinct training settings and data, rather than comparing with other alignment methods. This makes it difficult to infer the superiority of the proposed method over existing alignment techniques, as the comparisons are not directly relevant. 3. Clarity and organization: The writing is not easy to follow and requires improvement. Specifically, sections 4.3 and 4.4, which are parts of the methodology, are incorrectly placed in the experiment section. Additionally, the evaluation protocol (section 5.1) should be introduced in section 4 but is instead placed in the results section. This misplacement affects the clarity and logical flow of the paper, making it harder for readers to understand the methodology and its evaluation comprehensively.

Questions

1. Why does DPH not require an SFT sampling and human labelling stage? In section 4.4, you mentioned SFT in your training pipeline. 2. What is the relationship between the proposed approach and the sentence "Would I Lie To You?" in your title?

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors include a discussion regarding the limitations of the proposed method.

Summary

This paper introduces Direct Preference Heads (DPH), a novel method for aligning language models (LLMs) with human preferences at inference time. DPH works by adding an auxiliary reward head to the LLM that learns to predict human preference scores for generated outputs. This allows the model to self-evaluate multiple candidate outputs and select the highest-scoring one, effectively pruning undesirable responses. The authors argue that DPH offers several advantages over traditional Reinforcement Learning from Human Feedback (RLHF) methods like PPO and DPO: * Inference-time alignment: DPH aligns the model at inference time, avoiding the potential degradation of reasoning abilities often observed with RLHF during training. * Lightweight: DPH requires only a single model to produce both responses and rewards, unlike RLHF which typically involves multiple models. The paper presents two objective functions for DPH, separable and contrastive, and demonstrates their connection to Conservative DPO. Experiments on GLUE, GPT4All, and RACE datasets show that DPH consistently outperforms both supervised fine-tuning (SFT) and DPO alone.

Strengths

* Novel approach: DPH offers a new perspective on LLM alignment by focusing on inference-time pruning rather than modifying the generation process itself. * Theoretical grounding: The paper provides a theoretical analysis of the objective functions and their relationship to cDPO, demonstrating robustness to label noise. * Strong empirical results: DPH consistently outperforms baselines on various tasks, showcasing its effectiveness.

Weaknesses

* Lack of statistical significance: The paper does not report error bars or statistical significance measures, making it difficult to assess the robustness of the results. * Missing the comparison to standard RLHF methods: the paper does not include comparison to the PPO baseline, which is the most commonly used technique. * The authors do not discuss other inference-time alignment techniques that exist in the literature.

Questions

* The paper uses different sampling strategies for SFT and DPH. It would be helpful to discuss the rationale behind these choices and explore the impact of different sampling strategies on DPH performance. * Sampling multiple candidate outputs at inference time can be computationally expensive, especially for larger LLMs. The paper doesn't explicitly address this cost, which could be a practical concern for real-world applications. Can you elaborate more on this in the paper?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

The paper focuses on a 551M parameter model. It would be valuable to see how DPH scales to larger models, as RLHF is known to be more effective for larger models. Comparing DPH to RLHF-aligned larger models would provide a more comprehensive evaluation. Also RLHF baselines on top of DPO need to be added.

Summary

Paper proposes Direct Preference Heads, which learns a reward prediction head using a pretrained model, without affecting the model's output distribution.

Strengths

Comprehensive evaluation - paper presented experimental results across a wide range of tasks (NLU, commonsense reasoning, reading comprehension, etc. ), and compared the proposed method against a range of different baselines (Pretrained model, SFT, DPO, etc. ) Good presentation - results are cleanly presented, training objectives are clear Significance - results showed that the proposed method outperforms comparable baselines in most presented tasks

Weaknesses

Some missing/confusing parts about the proposed method: - How are the learned reward predictions used? Some parts of the paper seems to suggest that multiple responses are sampled from the model, and the reward head is used to rank the samples? I think this is an important component of how the proposed method would be used, so it would be great to further elaborate on this procedure, e.g. how many samples? - Some parts of the paper seems to suggest that the proposed approach only learns a reward prediction head on top of an existing model without modifying the original model (e.g. in abstract, "... without directly affecting the output distribution of the language modeling head ..."), but other parts of the paper stated that it is better to finetune the entire model when learning to predict rewards (e.g. Section 4.3). However, the stated reason for why DHP might be better than other RLHF techniques like DPO is that further alignment (past SFT) can hurt model performance. If DHP also requires further finetuning, what is the intuition for why DHP performs better than DPO? Related works - Paper introduces PPO, DPO and cDPO. However, there has been a large number of recent works in RLHF beyond these papers. It would be great to present a more complete related works section with a more comprehensive survey of existing work.

Questions

See weaknesses

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The authors propose an inference time method to align language models with human preferences without harming the model’s reasoning abilities. The method creates an auxiliary reward head that operates during inference to score potential outputs without changing the output distribution directly. The author validates approach by comprehensive evaluations on NLU, commonsense reasoning, and reading comprehension datasets.

Strengths

1. Theoretical Insight: The paper provides a theoretical analysis, connecting DPH to cDPO, and providing proofs to support the convexity and effectiveness of the proposed loss functions. 2. Novel Approach: The paper introduces a novel approach allowing for preference-aligned model fine-tuning without directly affecting the output distribution, potentially reducing negative side effects like hallucination and preserving the model’s original reasoning capabilities.

Weaknesses

1. Inference Time Overhead: The Direct Preference Heads method involves using language models to generate multiple candidate responses, which must then be evaluated for selection. This process is less efficient compared to traditional fine-tuning methods due to the additional computational steps required. 2. Dependence on Initial Sample Quality: The effectiveness of the generation-then-reranking approach highly relies on the quality of the initial responses generated. If the sampled responses are of low quality, DPH is unable to enhance the output, as it can only rerank the given candidates. 3. Limited Evaluation Scope: The evaluation of the DPH method is restricted to a single model of 550M parameters. It is not clear how well the method would perform when scaled to larger models, as the results might not be consistent across different model sizes.

Questions

1. The paper would be more convincing if the author applied the proposed method to a larger-sized model and demonstrated a scaling trend. The use of only one model makes the effectiveness of the approach a bit underdetermined 2. How does the proposed DPH method compare with other inference time algorithm such as rejection sampling? 3. How do you choose baselines when reporting performance? For example, Table 1 reports the model performance on BERT, an early-stage masked language model, instead of other competitive baselines.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed all the limitations in the conclusion section.

Response

I thank the authors for the response! It would be great to incorporate the clarifications in the rebuttal into the paper.