Deep Bayesian Active Learning for Preference Modeling in Large Language Models

Summary

This work proposes BAL-PM, a stochastic acquisition policy aiming to select informative data samples (i.e., the prompt and its corresponding paired responses) that need to collect human feedback for LLMs' preference learning. Specifically, BAL-PM mainly addresses the issue of sampling redundant data points in previous methods that rely solely on epistemic uncertainty, such as BALD. On top of the previous BALD acquisition method, BAL-PM employs a task-agnostic entropy estimator for the prompt distribution to enhance the diversity of acquired data samples. Experiments on the Reddit and CNN/DM datasets under a pool-based setting demonstrate the effectiveness of BAL-PM over random sampling and the BALD baseline.

Strengths

- The paper is easy to follow. - The method is well-motivated by the observation of the tendency of sampling redundant data points in previous methods. - The effectiveness of key components of the proposed BAL-PM is well justified and supported by the experimental results and ablation.

Weaknesses

- The experiment part is not compelling and clear enough to me. Regarding the reported log-likelihood metric, it is unclear to me how it is calculated (e.g., on preferred response?) and why it is the only metric used for evaluating preference modeling. It seems to me that it can not measure to what extent the chosen response is preferred over the rejected one. Regarding the datasets evaluated, the task is limited to text summarization. The experiment part can be improved by adopting more metrics on more general-purposed datasets/tasks to illustrate the effectiveness of the proposed method. - While encouraging data diversity is plausible and effective in LLM's preference learning according to the experiments, the necessity of using the feature of the base LLM to measure the entropy of the prompt distribution is unclear to me. Although it is mentioned as a common approach in lines 97-99, the motivation for focusing on feature space is still somewhat vague to me and may introduce extremely high computational overhead in the context of LLM's preference learning. - The proposed BAL-PM, which requires maintaining/updating an ensemble of adapters for collecting each batch, may be difficult to practice in large scale. - The discussion of prior works is inadequate. Besides the difference in configuration, the technical parts of other related active preference optimization of LLMs should be discussed more.

Questions

- What is the advantage of adopting the (static) feature space of the base LLM for estimating the entropy of the prompt distribution? Is there any evidence to demonstrate that it is superior to other approaches, such as using smaller proxy models (e.g., BERTScore) or simpler fashions? - I am curious about whether different adapters can make diverse predictive distributions with limited expressive power. Is there any empirical evidence for this?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

  The authors present a method, Bayesian Active Learning for Preference Modeling (BAL-PM) that seeks to learn a preference model in a sample-efficient manner within an active learning setting. A key insight by the authors is the consideration of task-dependent and task-agnostic uncertainty to encourage diversity in the sampled prompt-completion pairs. My main concern with the approach is the link between the pool-based active learning setting and active learning over an open-ended and continuous prompt space which I will describe below. As such, my recommendation is borderline pending the authors rebuttal on this point.  

Strengths

  The paper is well-written and the idea behind BAL-PM is well-motivated. The empirical results appear compelling in the pool-based setting and furthermore, the authors provide the code to reproduce their results. The analysis of the diversity of the acquired prompts is a useful diagnostic to validate the authors' hypothesis that BAL-PM attains performance improvements by encouraging diversity in the sampled prompts.  

Weaknesses

  __MAJOR POINTS__   1. My main concern with the current work is the link to the real-world problem of constructing a preference model in a sample-efficient fashion in the absence of a pool of labelled prompt-completion pairs. Specifically, as I understand, one of the main motivations for being sample efficient in constructing the preference model is the cost of human labelling. In the experiments considered by the authors, the datasets already contain labels. As such, I would ask a) what is the motivation for being sample-efficient in the pool-based active learning setting? b) assuming the authors feel performance on the pool based setting is representative of an open-ended active learning problem, how would they go about verifying that BAL-PM performs well in the non pool-based setting?   __MINOR POINTS__   1. It would be great if the references appeared in numbered order. 2. There are some missing capitalizations in the references e.g. "Bayesian". 3. The arXiv identifier is missing for reference 2. 4. When referencing PyTorch, [1] should be used rather than the earlier workshop paper version. 5. It would be great if the authors could provide instructions for reproducing the experimental results in the README of their GitHub repository. 6. Reference 7 was accepted at TMLR. 7. Line 38, I would not expect the statistic that agreement rates among human labelers is typically 60-75% to be robust across all problem settings. It would be worth qualifying this statement. 8. The arXiv identifier for reference 18 is missing. 9. "Prompt-completion pairs" would potentially be a more appropriate terminology in place of "prompt-answer" pairs. 10. On line 105, S is not defined, this should presumably be \Chi. 11. Line 134 typo, "an unsupervised". 12. In Figure 4, it would be great to include in the caption the number of random trials for which the errorbars are computed. 13. In the abstract the authors do not mention that the figures of 33% and 68% apply to a random sampling baseline.   __REFERENCES__   [1] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L. and Desmaison, A., 2019. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32. [2] Yang, A.X., Robeyns, M., Coste, T., Wang, J., Bou-Ammar, H. and Aitchison, L., 2024. Bayesian reward models for LLM alignment. arXiv preprint arXiv:2402.13210.  

Questions

  1. In Section 4, the authors mention that their Bayesian preference model is constructed using an ensemble of adapters. What was the motivation for this choice over Laplace-LoRA [2] for example? 2. In Figure 7, random sampling outperforms BAL-PM for the 140b parameter model early in the active learning trace. Do the authors have an explanation for why this might be the case? 3. It would be interesting to see the full details of the validation procedure for choosing the hyperparameters, most notably the entropy term \Beta. How sensitive was performance to the choice of \Beta hyperparameter? What is the link to Section F of the appendix? 4. In terms of the results presented in Figure 9, how do the authors compute the aleatoric uncertainty for the datasets? Are these uncertainty estimates provided in the datasets?  

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

  The biggest limitation I foresee with the current work is the potential for the performance of BAL-PM to carry over between the pool-based active learning setting and the non pool-based setting as described above.  

Summary

This paper proposed a novel framework to select the most informative preference data for training based on Bayesian active learning. To collect the prompt-response pair (x,y), it firstly selects the prompt based on Bayesian Active Learning by Disagreement by maxmizing the information gain. Then, the selection of response considers both the preference model epistemic uncertainty estimation and entropy estimation for the acquired prompt distribution, which addresses the challenges of selecting diverse samples. When evaluated on Reddit and CNN/DM preference datasets, it saves 33% and 68% training samples by comparing to random sampling.

Strengths

1. The research problem is important to data-efficient training. 2. The presented emprical results are impressive in saving utilised training data over listed baselines. 3. The empirical studies is comprehensive and verify the effectiveness across different settings.

Weaknesses

1. Lack for diversity-based and uncertainty-based sample selection methods, such as [1,2,3,4], which makes the technical contributions less reliable, 2. As the method is based on feature extraction from existing LLMs, the computation cost in estimating the diversity and uncertainty is not clear, which inhits its pratical application. References: [1] What Makes Good In-Context Examples for GPT-3? [2] An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels [3] Demystifying Prompts in Language Models via Perplexity Estimation [4] Diverse Demonstrations Improve In-context Compositional Generalization

Questions

See the details in Weakness. 1. How are the performances of existing other baselines. 2. What are the computation cost in estimate the utility score, diversity and entropy.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors mentioned the intrinsic limitation of the transformer's embedding, i.e., noise-TV. The impropriate selected samples can be toxic, unfair or not well-represented.

Summary

This paper presents BAL-PM, a Bayesian active learning framework for the training preference model. The authors propose an acquisition policy that seeks examples that have high epistemic uncertainty and can maximize training data’s entropy. Specifically, the epistemic uncertainty is estimated by the training preference model and entropy is estimated by the base LLM. Experiments on two preference datasets show that the proposed method requires fewer training data compared to previous methods (33% and 68% reduction on each dataset accordingly). The author also shows that their method is scalable to larger LLMs.

Strengths

- The problem is clearly formulated. The method is well derived and justified. - The results are strong, showing significant improvement in sample efficiency. - The authors test their method on models of various sizes, demonstrating its scalability. - The paper is clearly written and easy to follow.

Weaknesses

- The evaluation metric is limited. The authors only show the log-likelihood of the test sets. This single metric has no guarantee that preference models trained by this method have a more accurate judgment (shown by pair preference accuracy) and lead to better fine-tuned LMs (shown by tuning with the trained preference model)

Questions

- How do the results compare to models trained on the whole dataset? - Do the trained models lead to better LMs?

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have properly adequately the limitations of their work in the paper.

Reply to the Rebuttal

The authors did a good job of addressing most of my concerns and making clarifications regarding their method and evaluation. However, I strongly encourage the authors to validate the effectiveness of BAL-PM on a broader range of tasks/datasets, especially given the notable computational efficiency of the proposed method and the moderate scale of common academic preference datasets. In my humble opinion, the current paper is a nice technical contribution with fair empirical significance to the community.

Many Thanks to the Authors for their Rebuttal

  Many thanks to the authors for their rebuttal. 1. In terms of the pool-based vs. non-pool-based setting, I think it would be beneficial to include a preference simulator for the non-pool-based setting which would a) emulate an open-ended active learning loop b) allow some simulated notion of ground truth aleatoric uncertainty to be present. I would encourage the authors to consider this for the camera-ready version of the paper. 2. Many thanks for the clarification on the use of adapters in place of Bayesian LoRA. The intuition regarding finetuning the base LLM makes a lot of sense and I can see why this would be an important factor to stabilize the surrogate in an active learning loop. 3. The intuition for the performance of BAL-PM relative to random sampling early in the trace makes a lot of sense. 4. Many thanks for the sensitivity analysis on the \Beta parameter. Given the above, I am happy to raise my score but I would strongly encourage the authors to think about including an additional experiment emulating an open-ended active learning loop with a preference simulator. I think this would "complete" the paper in the sense that both a) the pool-based setting with realistic human preferences and b) the non-pool-based setting with synthetic preferences are considered.