Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.
Paper
Similar papers
Peer review
Summary
This paper approaches the challenge of pluralistic alignment, which arises when the preferences among humans diverge across a population. The work identifies that current preference modeling assumptions do not account for multi-modal reward distributions, which is often the case for pluralistic preferences. Thus, they end up averaging modes and generating inaccurate reward models. To address that, the paper proposes a latent variable model that explicitly models different users and applies variational inference techniques to infer the user latent that conditions the reward model and the optimized policy. The paper also proposes scaling rewards following the likelihood from the model, as a way to ensure all latents follow the same scale, which is necessary for latent-conditioned policy optimization. Finally, the paper evaluates the proposed model in a set of control and language environments, demonstrating gains over prior preference modeling choices.
Strengths
- The paper approaches pluralistic alignment, a challenge that is mostly always overlooked by the RLHF community, which implicitly assumes that all humans present the same preferences and values. Therefore, the work is very relevant; - The main merit of the paper is to identify that current modeling assumptions, while assuming the same preference over humans, fall short when modeling diverse preferences. The paper brings very didactic cases that explicitly show such failure and the consequences in the optimized policy (e.g., Figure 3). - The introduction of the latent variable modeling for preference modeling in RLHF is interesting and novel, and a natural extension of the prior work [1].
Weaknesses
- The major concern is on the evaluation setup. Although the paper brings a diverse set of experiments, they are all setups where the hidden context is composed of very few variables with simple distributions. This is acceptable to show the failure of previous methods, but not sufficient to claim scalability in the LLM setup for the proposed method. In more detail: - The “Pets” dataset is too simple and does not even require language. From the example in the Appendix B.2, it is possible to ignore the prompt and extract the pet variable from the responses and context. The problem then boils down to fitting a categorical classifier over four classes, where the input is the pet in response A and B, and the same for the context. Given the amount of variables and the dataset size, it is questionable if variational inference is required here - perhaps a simple linear model or MLP can learn directly from the context. - The “Ultrafeedback-P” indeed contains much richer prompts and responses, and the hidden variables (helpfuness, honesty) are not straightforward to extract from the responses. However, the experiment assumes only two users, and there are only two hidden variables as well. Again, this raises concerns if variational inference is required here - perhaps a base LLM can already separate these users in the feature space, but there is no such a baseline in the work. - Interestingly, the paper highlights the requirement of filtering out the context where the users agree, which simplifies the task, as it is easier to identify users with extreme disagreement. It would be important to justify why that is necessary and if it is not a limitation of the proposed method. - The paper is motivated by personalized RLHF and provides modeling for latent-conditioned policies, but in the LLM experiments there is only preference modeling and no RLHF/alignment. This time of experiment is necessary to claim that VPL scales up for pluralistic alignment. - Some methodology details are missing: how the prior is learned (the training objective, data); how humans are simulated in the control environments (L279); how the accuracy of the reward model in Table 1 is defined; the description and details of the environment in used in Appendix A.3; - Another crucial concern is that the proposed reward scaling technique is not equivalent to the learned latent reward in the sense of resulting in the same optimal policies (it is not a policy invariant reward shaping [2]). A minimal example that illustrate this is the following MDP: - Imagine an MDP with 11 states (s_0 to s_10). Initial state s_0. There are two actions, a_1, a_2. If the policy takes a_1, it goes to s_1 and receives r_1 = 1000. If it takes a_2, it goes to s_2 and receives r_2 = 100. From s_3, regardless of the action taken, the agent goes to s_1 -> s_3 -> s_5 -> s_7 -> s_9 (terminal), and does not receive rewards in any of these states. Similarly, from s_2, it goes to s_2 -> s_4 -> s_6 -> s_8 -> s_10 (terminal), but receives reward +100 at each state. Assuming no discount in the return, we have V(s_1) = 1000 and V(s_2) = 500, thus the optimal policy should choose a_1. However, if you consider the reward scaling proposed in this work and assume a uniform distribution over states to compute the expectation in L211, then V(s_1) ~= 17/9 and V(s_2) ~= 30/9, which is a different optimal policy. - The reward shaping seems to work empirically at least for the Maze Navigation (Ravens-Manipulation is slightly worse and it is unclear in the LLM experiments if the VPL version leverages the scaling or not). It is important to state this caveat, since reward scaling implicitly assumes policy invariance under reward transformation. - One minor concern is regarding the data requirements during test-time inference. During deployment, VPL would require interacting with a user to collect a few labels before performing inference. In the LLM experiment with GPT2, it looks like it requires 100 samples, so it is questionable if this is sample-efficient enough for interacting with humans in the real world. Is this experiment using the active inference from Section 4.2? **Summary**: I believe the paper brings a contribution while presenting the failure of previous methods in pluralist alignment, and showing how they fail. Furthermore, the proposed methodology is interesting and seems to work well in small cases. Nevertheless, the presented empirical evidence does not support the scalability claims in LLMs, as the datasets are too simple and even raises questions whether very simple models could solve it. Furthermore, the reward scaling technique proposed is not well grounded theoretically and leads to different optimal policies. There are also some potential societal consequences that are not discussed. Therefore, I do not support acceptance in the current form of the paper, but. I am open to changing my score in case of the raised concerns being addressed or any misunderstanding clarified.
Questions
- How does VPL handle irrational users? (i.e., users whose preferences are inconsistent across context/responses) - Do the LLM experiments leverage the active inference from Section 4.2? - Why isn’t VPL-SPO shown in Table 1?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
- I believe the experimental setup does not support any claims of scalability with LLMs and it is unclear whether the proposed method would work in a scenario with multiple users and several variables affecting preferences, which is often the case in the preference datasets. - There are some concerns in terms of social impact that are not discussed in Section A.5. A well-performant latent model results in a very personalized language policy, which can potentially lead to two main problems: first, the model being biased/presenting sycophancy [3] to satisfy user beliefs; second, the model reinforcing users with antisocial/unethical/criminal preferences, which often presents preferences that are very far from the rest of the society. Both scenarios are potentially harmful and would be interesting to discuss ways to mitigate them. **References**: [1] Siththaranjan et. al. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, 2023. [2] Ng et. al. Policy Invariance under Reward Transformations: Theory and Applications to Reward Shaping. ICML, 1999. [3] Sharma et. al. Towards Understanding Sycophancy in Language Models, 2023. # Post-Rebuttal Please refer to my comment under the rebuttal message. Based on that, I am increasing my scores and recommending acceptance.
Continued Rebuttal
> **the model being biased/presenting sycophancy [12] to satisfy user beliefs; second, the model reinforcing users with antisocial/unethical/criminal preferences, which often presents preferences that are very far from the rest of the society.** Thank you for bringing up this insight, and we will include the following discussion in the social impact statement. In pluralistic alignment, we assume that some differences in user preferences reflect divergent but equally valid perspectives on which moral or cultural values an LLM should align to; for example, individuals from one culture may hold collectivist moral values, while another culture is more individualist. Both value systems should be respected, and as such LLMs should be able to recognize and adapt to the values held by a particular user. However, the personalized model could potentially either be sycophantic or align with adversarial users, which is undesirable. This raises very interesting questions, such as: At what point should the LLM embrace a more universal set of values? How can we detect when such a point has occurred in a particular conversation? The probabilistic framework of the user distribution could allow us to identify low probability or rare behavior, and also the distributional nature of reward functions can help us point out responses where the users are divergent (maybe signifying disagreement). Additionally, a model could flexibly switch between adhering to the user's personal preferences and conforming to a more universal perspective on topics where it could be biased, or is sensitive to jailbreak [14]. Taking inspiration from Constitutional AI [14], we can allow a system designer to specify the topics for which the LLM should not engage in user personalization. Overall, this presents an exciting future research direction toward building safe personalized LLMs. > **Pets dataset is simple** We acknowledge that **Pets dataset is a simple synthetic dataset**. Therefore, we present the improved capabilities of our method with additional experiments on a larger and more diverse UltraFeedback dataset (UF-P-4), with four different users (In Table AM:1). The Pets dataset provides a **sanity check over VPL** to show that the model is able to adapt to multi-modal preferences in imbalanced language datasets. > **“this raises concerns if variational inference is required here - perhaps a base LLM can already separate these users in the feature space, but there is no such a baseline in the work.”** We include an additional baseline in Table AM:1, "VPL+Ground Truth User", where we replace the predicted latent user vector by the ground truth one-hot user vector to adapt the model to different users at training and test-time. **We show that VPL provides comparable performance to this condition (63% vs 66%)**. Here, a key advantage of VPL is that **it does not assume access to explicit user types** but learns to encode and cluster users directly from preference labels, while achieving similar accuracy. > **there is only preference modeling and no RLHF/alignment** We present experiments on learning policies using VPL reward modeling in simulated control domains. We acknowledge that our language experiments are focused on preference modeling, but we primarily follow and compare to prior work, which also focuses on preference modeling from diverse datasets alone [2, 3], without including experiments for LLM fine-tuning. We believe that preference modeling alone is interesting, as modeling diverse users may also provide increased interpretability in highlighting potential reasons for preferences [7, Sec. 2]. We believe that further exploring how to best apply VPL to downstream tasks and larger, noisier datasets is an interesting and exciting avenue for future research. Additionally, prior work has shown that improving RM performance can yield improved downstream performance, both when used in RLHF training [8, 9, 11] and when used in best-of-N settings [9, 10]. > **“the requirement of filtering out the context where the users agree..”** In the additional experiments with four users (Table AM: 1 of the rebuttal PDF), we filter out instances only where all users disagree. So, the context can still contain queries where at least two users overlap. Thus, **VPL works in cases where different users agree on some responses, but not all of them.** We will add the following to the limitations sections of the paper: “In LLM preference modeling, VPL assumes that the queries used to generate context inputs for posterior inference contain some useful information to identify the users. In our work, we filter out the instances where all users agree i.e. avoiding degenerate contexts that provide no information about the users.”
Continued Rebuttal
> **"Methodology details"** Thank you for pointing out the missing details. We will update them clearly in the manuscript, as follows: 1. We assume that our prior is a multi-variate Gaussian with mean $\mu$ and covariance $\Sigma=\text{diag}(\sigma\sigma^T)$, where $\mu, \sigma \in \mathrm{R}^d$. In all experiments, they are initialized from a standard Gaussian. However, in our control experiments, we observed that using a learned Gaussian i.e. setting $\mu$ and $\sigma$ to learnable parameters under the ELBO objective (Eq. 3) improved performance and stability during training. 2. The humans are simulated using Oracle reward functions that are included in Appendix B.1. We randomly sample a user type and query a batch of annotations from the given user to generate a single data point. 3. The accuracy of the reward model is "1" if the LM rewards model assigns a higher reward to the response preferred by the given user over the alternative response, and "0" otherwise. We report the average accuracy of the model over the eval set consisting of multiple prompt and response pairs, labeled by diverse users. 4. VPL-SPO is required as a part of policy optimization while currently, we focus on preference modeling for LLMs in this work. VPL and VPL-SPO will model the preferences with similar accuracy. The only difference would be the scale of rewards that affects policy optimization. -- [1] Swamy et al. (2024). A Minimaximalist Approach to Reinforcement Learning from Human Feedback. [2] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF [3] Zhao et al. (2023). Group Preference Optimization: Few-Shot Alignment of Large Language Models. [4] Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. [5] Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. [6] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. [7] Sorensen et al. (2024). A Roadmap to Pluralistic Alignment. [8] Shen et al. (2023). The Trickle-down Impact of Reward (In-)consistency on RLHF. [9] Gao et al. (2022). Scaling Laws for Reward Model Overoptimization. [10] Ivison et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. [11] Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. [12] Sharma et al. (2023). Towards Understanding Sycophancy in Language Models. [13] Ouyang et al. (2022). Training language models to follow instructions with human feedback. [14] Bai et al. (2022) Constitutional AI: Harmlessness from AI Feedback.
I appreciate the authors’ efforts on addressing questions and bringing new empirical evidence for the work. My major concern (about the empirical evidence not supporting the scalability claims) was properly addressed with the rebuttal experiments. There are now new experiments showing the method works for >100 users in Habitat Environment and an extension in the Ultrafeedback benchmark. The missing methodology details were also clarified in the rebuttal. I strongly recommend the authors to add these experiments and new clarification text for the camera-ready version, as it considerably improves the paper. My third concern was also clarified. In terms of the second concern (reward scaling), the rebuttal acknowledges the limitation and links it to an underlying assumption on the oracle’s preference labeling process (non-Markovian). I believe this assumption is strong and somewhat simplistic (if the ground-truth labeler only looks at the trajectory’s return and not for particular states, then the problem modeling could be simplified to a bandit setting instead of the full sequential decision-making setup - as it is often the case of traditional LLM-based RLHF). I believe the camera-ready version should also explicit this limitation/underlying assumption as a clarification. Overall, most of my crucial concerns were addressed and the paper largely benefited from the new content. Therefore, I am raising my score accordingly. Other points (questions/limitations) were properly discussed as well. Again, I strongly recommend authors to incorporate this on the camera-ready version, particularly the discussion regarding the societal impact. (For the next time, **please make sure to adhere to the rebuttal length restriction**).
Summary
The primary objective of the paper is to design a multi-modal RLHF strategy to align diverse preferences with a latent variable model. The latent variable represents the users/topics and the reward model conditioned on the latent variable is learned for each user preference. The empirical results support the hypothesis on simulated control problems and pluralistic language datasets.
Strengths
1. Aligning to diverse preference with variational inference is one of the most natural ideas and the work provides an interetsing step in that direction. 2. The approach also provides a method to actively learn user preferences leveraging the posterior uncertainty. 3. The empirical performance and ablation shows that learning under the probabilistic framework is able to precise multimodal reward model.
Weaknesses
1. The setup is not extremely practical. For ex: If I am the LLM company when a new user comes to the system and asks a question via the prompt? Then, instead of answering this, we will be doing some active learning to efficiently identify the user right? Since the posterior q(z|y_1, y_2) needs to be estimated for that? Is there any other way to do that or we need to incorporate the active learning strategy every time for each new user/group? 2. The experimental setup is restricted to a simulated environment and tasks. It is crucial to understand the efficacy of the approach when scaled to more realistic environments and tasks [1, 2, 3] 3. The work lacks comparison with several baselines on multi-objective RL or aligning with diverse preferences. Please provide a detailed comparison with them, not necessarily through experiments but at least a detailed discussion will be crucial. [1, 2, 3, 4, 5] References : 1. MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences 2. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation 3. Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration 4. Pareto-Optimal Learning from Preferences with Hidden Context 5. PERSONALIZED SOUPS: PERSONALIZED LARGE LANGUAGE MODEL ALIGNMENT VIA POST-HOC PARAMETER MERGING
Questions
1. Can the authors motivate a practical scenario of the setup, where this setup can make sense? 2. It's not very clear on the specific nature of the prior and posterior being used for each of the tasks. 3. Also it is done on majorly simulated tasks and environments. Hence, it is important to show the performance in more realistic environments and benchmarks.
Rating
4
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
Check above
Continued Rebuttal
> **“comparison with several baselines on multi-objective RL or aligning with diverse preferences. ”** We thank the reviewer for providing additional references for our work. We will incorporate the additional references in the related works section as follows: [1, 4] aims at trading conflicting alignment among diverse users with different objectives through techniques like Pareto-optimal optimization or multi-objective RL. The goal of such methods is to optimize the reward model to maximize worst-case performance over the different groups. In contrast, **our work does not aim to optimize against the diversity but rather solve the model misspecification** and learn reward models that can infer the context and specialize to a particular user. This ensures the model can align to all user groups, rather than trade-off among them. [2] introduces an approach that uses explicit clustering of human groups and learns individual reward functions for the different clusters. Our work instead relies on variational inference and latent conditioned rewards to infer and model diverse humans directly from the preference data. [2] further introduces an additional method that assumes a single reward function for all the clusters, but **adopts a probabilistic approximation to the reward model, similar to DPL [6]. We include DPL as a baseline in all the LLM experiments** in Table 1 in the original submission and Table AM:1 in the rebuttal PDF, and show that our method outperforms DPL across multiple datasets. We included a reference to [5] in L112 of the original submission. [3] proposes a pluralistic alignment framework, **using smaller LMs that are trained on community or user-specific data**. Further, it uses responses from the smaller community LMs to adapt a larger LLM to provide responses covering all or just one specific user. Meanwhile, **our approach adopts an unsupervised method (no access to explicit user distributions)** to identify the latent and condition the preference model towards the specific user preferences. > **prior and posterior structure** Here, we provide additional details about the structure of the prior and posterior of our model. We assume that our prior is a multi-variate Gaussian with mean $\mu$ and covariance $\Sigma=\text{diag}(\sigma\sigma^T)$, where $\mu, \sigma \in \mathrm{R}^d$. In all experiments, they are initialized from a standard Gaussian. However, in our control experiments, we observed that using a learned Gaussian i.e. setting $\mu$ and $\sigma$ to learnable parameters under the ELBO objective improved performance and stability during training. The posterior is an MLP that takes in the annotated samples $\textbf{S} \sim (s^i_1, s^i_2, y^i)_{i=1}^N$ and predicts the latent distribution $\mathcal{N}(f(\textbf{S}), g(\textbf{S}))$. > **Also it is done on majorly simulated tasks [...]** Our work presents **an algorithmic solution to the problem of personalization** in RLHF. We present extensive experiments across simulated control and language experiments, which we believe present realistic setups and benchmarks. So, we believe real-robot experiments to be beyond the scope of this algorithmic paper and leave on future work the challenge of deploying this method to real-world robot systems. -- [1] Chakraborty et al. (2024). MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences. [2] Park et al. (2024). RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation. [3] Feng et al. (2024). Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration. [4] Boldi et al. (2024). Pareto-Optimal Learning from Preferences with Hidden Context [5] Jang et al. (2024). Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. [6] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF [7] Wu et al. (2023). TidyBot: Personalized Robot Assistance with Large Language Models. [8] Graham et al. (2013). Moral Foundations Theory: The Pragmatic Validity of Moral Pluralism. [9] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. [10] Bhattacharjee et al. (2020). Is More Autonomy Always Better? Exploring Preferences of Users with Mobility Impairments in Robot-assisted Feeding.
Summary
Instead of learning a unimodal reward model as in standard RLHF, this work aims to learn a reward that covers a diverse range of preferences. It assumes that user preferences are not explicitly given, such as through verbal descriptions in the prompt/instruction. Instead, preferences are implicitly provided through rankings among candidate responses. Technically, it uses variational inference to learn an encoding that characterizes any user’s preferences. Accordingly, the policy model is conditioned on the learned preference latent code, making the generation steerable. The authors also discuss how to select the most representative set of response pairs for each user.
Strengths
- The paper is well-written and easy to follow. - The study is addressing a crucial problem. The setup is realistic, as users may not always want to explicitly state their preferences.
Weaknesses
1. It is unclear how many labels are needed to accurately learn a user preference or profile encoder. There should have been experiments evaluating (a) how encoder’s performance improves as the # of user samples increases; and (b) how well the encoding generalizes — can it encode unseen user profiles with high fidelity, and can it extrapolate and interpolate? 2. The claim of being the first work that learns latent-code conditioned reward is not correct. [1] and [2] below also learn multi-modal reward. - [1] Guan, Lin, Karthik Valmeekam, and Subbarao Kambhampati. "Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences." ICLR 2023 - [2] Wu, Zeqiu, et al. "Fine-grained human feedback gives better rewards for language model training." NeurIPS 2023 3. The setup of the LLM experiment is quite simple -- the number of attributes or dimensions of preferences is very limited. While the feasibility of attribute-conditioned reward modeling has been demonstrated in previous works [1,2], this work doesn't significantly extend beyond them in terms of scalability. 4. One missing piece in the experiment is an analysis of the latent code's negative impact on the language model. An important feature of language models, especially LLMs, is their versatility. They are not supposed to only answer questions related to pets. One question that needs to be answered is whether conditioning on a pets-related latent code would lead to catastrophic forgetting or distortions in responses to other tasks/questions unrelated to pets. 5. I understand that Section 4.2 discusses the strategy for selecting the most representative state-pair sets during deployment. However, this process seems quite costly as it requires multiple full passes through the dataset. A complexity analysis should have been included. 6. Real-world user profiles and preference data are often unbalanced. The imbalance may affect the approach's effectiveness.
Questions
See the Weakness section.
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
See the Weakness section.
Continued Rebuttal
> **“This process seems quite costly as it requires multiple full passes through the dataset. A complexity analysis should have been included.”** Thank you for this comment, we would like to clarify **that our approach doesn't require multiple passes over the entire dataset for each new user**. In our active inference techniques, we use a sampling-based method inspired by [3] to actively query the model. Given a dataset of D queries $(s^i_A, s^i_B)_{i=1}^{|D|}$, we sample $S$ query batches of size $Q$, where $Q$ is number of annotated samples per batch we get from a user. Here, $Q \in [2,8]$, **so we need to perform O(S * Q)** passes over the model with batch size $2^Q \sim [4, 256]$. Furthermore, this process only needs to be performed once after the model is trained to obtain the most discriminative set of queries for the given model. Finally, whenever a new user interacts with the system, we need to get labels on the actively inferred queries (usually 2-4) but do not require any additional passes over the query dataset. > **“The imbalance may affect the approach's effectiveness.”** Thank you for your attention to this detail. In our control and language experiments, particularly the control experiments and the pets dataset are imbalanced i.e. the preference dataset contains an unequal number of preferences from individual groups or users. As a result, the baselines can achieve > 50% accuracy, converging to the preferences of the majority user groups. So, **VPL works in the presence of imbalanced datasets.** While recent works [7, 8] focus on achieving Pareto-optimal performance across the groups, VPL treats each user individually via latent conditioning and does not suffer from this problem. VPL is able to personalize to the minority groups as well (see Figure 11 in Appendix A). > **One missing piece in the experiment is an analysis of the latent code's negative impact on the language model. An important feature of language models, especially LLMs, is their versatility. They are not supposed to only answer questions related to pets. One question that needs to be answered is whether conditioning on a pets-related latent code would lead to catastrophic forgetting or distortions in responses to other tasks/questions unrelated to pets.** Thank you for raising this insightful question. VPL introduces a latent bottleneck during the reward learning process, but since the base architecture of the policy and the reward model does not change otherwise, we do not believe this should majorly affect the performance of the model. We will leave a thorough investigation of this question to future work, but will note this as a potential limitation in the limitations section. -- [1] Guan et al. (2023). Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences. [2] Wu et al. (2023). Fine-grained human feedback gives better rewards for language model training. [3] Sadigh et al. (2017). Active Preference-Based Learning of Reward Functions. [4] Peng et al. (2024). Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input. [5] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. [6] Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. [7] Boldi et al. (2024). Pareto-Optimal Learning from Preferences with Hidden Context [8] Chakraborty et al. (2024). MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences.
I thank the authors for their detailed response. First, I would like to point out that the rebuttal exceeded the character limit by being posted as an Official Comment instead of as a Rebuttal. I hope the authors can better adhere to the conference policy and respect the time of reviewers. While I don't think the rebuttal adequately addresses my concerns, I still find that the upsides of this work outweigh the downsides. Therefore, I would like to maintain my current positive rating.
We would like to thank the reviewer for their response. We apologize for the response length and will be considerate of that in the future. Could the reviewer please point us to specific questions to expand upon? This would help us address their concerns more effectively.
Summary
This paper introduces a new framework for preference learning which tailors to user-preferences. Human feedback with Variational Preference Learning (VPL) learns a latent reward / preference for each user at the test-time. They furthur show potential application of techniques from active learning and uncertainty estimation to the framework.
Strengths
- Paper is generally well-written and clear - The research problem is very well motivated -- personalization to preferences that go beyond a universal notion of a single preference function. - Method itself is novel and well-formulated to match the problem.
Weaknesses
- My main concern lies in evaluation outside of designed control environments. To truly test the quality of a Reward Model (RM), one needs to show the downstream policy performance benefiting from improvements in the RM. These results are not present currenlty in section 7. The issue of reward variance as discussed in Section 4.1, may require further design decisions coupled with the optimization algorithm (PPO, REINFORCE, RLOO, online contrastive losses like DPO, etc.). Previous work has studied the general issue of gradient variance in RLHF (which is directly related to reward variance through the REINFORCE estimator) which suggest that this may not be an issue [1]. [1] Ahmadian et. al. "Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs"
Questions
Suggestions: - Having a single-modal dataset, such as summarization, would help further ground the work and increase the experimental depth. The expectation is that we shouldn't see many benefits when using VPL compared to traditional RLHF.
Rating
7
Confidence
3
Soundness
4
Presentation
3
Contribution
3
Limitations
Limitations have been specified by the authors.
Continued: Author Rebuttal
> **Generalization to unseen users.(Reviewers KQbQ)** In Figure AM:3, we compare VPL against the baselines in a setting where the agent is trained on preferences from users preferring goals from a set of 10 locations in the maze, but at evaluation, **it interacts with users preferring goals from a set of 5 unseen locations** (sampled in distribution). We include a ground truth baseline, conditioned on the exact training and test locations (so explicit goal information). We show that VPL can interpolate between users and pursue the unseen goals better than the baselines, performing comparably to the ones with ground truth information. **This provides evidence that VPL could adapt and personalize AI models to unseen users.** > **VPL performance in unimodal settings (Reviewers PuPi)** An interesting concern by reviewer PuPi was to show the performance of VPL in settings with an unimodal preference dataset. We ran additional experiments on the UltraFeedback dataset, where we considered the preferences of a single user (preferring the model to be “honest” over all other attributes) to analyze the single-modal case as suggested. The standard BTL model gives a 77.04% eval accuracy while our VPL model gives a 77.16% eval accuracy. Our model matches the baseline performance, indicating that **there is no drop in performance when using VPL compared to traditional RLHF over an unimodal dataset**. > **Better explanation of the active learning approach to VPL (Reviewers hKQW, KQbQ)** Here, we include additional details to clear a misunderstanding regarding the active learning experiments. In our active inference technique, we use a sampling-based method inspired by [9] to generate the active queries for the model. Given a dataset of D queries $(s^i_A, s^i_B)_{i=1}^{|D|}$, we sample $S$ query batches of size $Q$, where $Q$ is number of annotations per batch we get from a user. Here, $Q \in [2,8]$, so we need to perform O(S * Q) passes over the model with batch size $2^Q \sim [4, 256]$. Furthermore, this process only needs to be **performed once after the model is trained to obtain the most discriminative set of queries for the given model.** Finally, whenever a new user interacts with the system, we need to get labels on the actively inferred queries (usually 2-4) but do not require any additional passes over the query dataset. In our experiments (Figure 5), we show that using active learning allows the model to achieve comparable performance with fewer queries (\~2), as compared to randomly sampled larger (\~8) queries. -- [1] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF [2] Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. [3] Zhao et al. (2023). Group Preference Optimization: Few-Shot Alignment of Large Language Models. [4] Conitzer et al. (2024). Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. [5] Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. [6] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. [7] Puig et al. (2023). Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots. [8] Wu et al. (2023). TidyBot: Personalized Robot Assistance with Large Language Models. [9] Sadigh et al. (2017). Active Preference-Based Learning of Reward Functions.
Response to Rebuttal by Authors
Thanks for providing clarifications to my concerns. I agree with the motivation for robotics, although for the LLMs still its not very clear. So, to be clear once you have the N pairs, with which you perform the active learning. But, what does it mean that the active learning is done once is not clear? Can you explain this in more detail. Also, can you please specify the prior and posterior used for the robotics and LLM case?
Thank you for engaging in discussion with us. We provide additional clarification regarding the motivation and practicality for VPL with LLMs. We also include a detailed explanation of the active learning approach and the prior/posterior structure (in addition to the outline in Section 4). ### ***LLM Motivation*** As we show in our new experiment for reviewer PuPi, by default our method gives the same reward modeling performance as the current standard BTL model if there is a single user in the dataset or if we have no additional information from the user. So, if we have no additional preference labels from a user, our technique will not hurt performance. **However, if we can obtain 2-8 annotations from a user about which response they prefer, we can personalize our reward model to their specific preferences and values, unlocking the benefits of pluralistic alignment of LLMs.** ### ***Active Learning*** We would like to clarify the active learning workflow in detail: 1. We have a set of queries $(s^i_1, s^i_2)^K_{i=1}$, which is a pair of states (or responses to a given prompt, in case of LLMs). To create the training set, we sample a batch of queries of size $N$, and randomly ask one of the users to annotate it with their preferences ie. we get $\mathbf{S_j} = [(s^i_1, s^i_2, y^i)_{i=1}^N]_j$, where $N << K$. 2. We get multiple annotated batches from the diverse users to form the training set for the reward model i.e $D = (\mathbf{S_1}, \mathbf{S_2} \dots)$. 3. We train the reward model as indicated in Algorithm 1, and obtain the encoder $q(z | \mathbf{S_j})$ (i.e it takes as input a subset of queries $\mathbf{S_j}$) and the reward model R(s,z). The encoder output is $q(z | \mathbf{S_j})$, which is multi-variate gaussian distribution approximating the distribution over user preferences / groups or types. 4. ***Here, we start the active learning process***. Given $q$ and all the queries $(s^i_1, s^i_2)_{i=1}^K$, we generate multiple subsets of size N (total possible samples are $^N C_K$). For each given subset, we can find the information gain over the posterior in Step 3 and Eq. 4. 5. We choose the subset of N pairs $\textbf{S}_{active} = (s^i_1, s^i_2)_{i=1}^N$ with the max information gain. This is the set of questions that are most informative about the user type or distribution. 6. Finally, **at evaluation for all incoming users, we ask them to annotate the same set of questions in $\textbf{S}_{active}$, and then provide it as input to the encoder to obtain the posterior over this user's preferences**. **Therefore, the entire process of active learning has to be done only once after training to obtain $\mathbf{S}_{active}$. At eval time, we just need a users to provide labels for the same $N$ pairs to predict the posterior distribution for reward / policy conditioning.** ### ***Posterior and Prior:*** 1. We use a gaussian prior and an MLP based posterior similar to the approach in a standard VAE [1]. 2. **The prior is a standard multi-variate gaussian of dimension** $d$, where the mean $\mu=[0 \dots 0]^T$ and the covariance $diag(\sigma \sigma^T), \text{ where } \sigma=[1 \dots 1]^T$. Here, $\mu, \sigma \in \mathrm{R}^d$; $d$ is the size of the latent dimension. In the robotics experiments, we set them to be learnable parameters by setting the requires_grad property = True in pytorch. 3. For, **the posterior in robotics and LLMs we predict two vectors** $\hat{\mu}, \hat{\sigma} \in \mathrm{R}^d$ **using the encoder q and the annotated pairs $\textbf{S}_j$ as input**. It generates the posterior, which is also a multi-variate gaussian with the mean $\hat{\mu}$, and the covariance $diag(\hat{\sigma} \hat{\sigma}^T)$ 4. The encoder architecture in the robotics case is a simple MLP that takes in the annotated pairs $\mathbf{S_j} = [(s^i_1, s^i_2, y^i)_{i=1}^N]_j$ i.e. the input is of dimension (2S+1)*N, where S is state dimension, and N is the number of queries. 5. For the LLM encoder architecture, we lay it out in detail in Section 5 and Figure 2 of the paper. It is similar to the robotics case, with certain modifications to handle the high dimensional and complex LLM embeddings. 6. For robotics, we sweep over possible values $d$ in { $8,16,32$ }, and for LLMs, $d=512$. We include the detailed hyperparameters in Appendix B.4 7. Finally, to predict the reward or condition the policy on the user type/preference, we sample a vector $z$ from the predicted posterior, and augment the input to the reward model or the policy to generate personalised rewards / behavior. [1] Kingma et al. (2013). Auto-Encoding Variational Bayes. Thank you for the suggestions and we will make this workflow more clear in the paper.
Decision
Accept (spotlight)