Panacea: Pareto Alignment via Preference Adaptation for LLMs

Summary

# Summary The paper presents Panacea, an innovative method for aligning LLMs with human preferences, by reconceptualizing the alignment task as a Multi-Dimensional Preference Optimization (MDPO) challenge to recover the entire Pareto front and adapt online. # Contribution Panacea marks a good advancement in LLM alignment by providing a scalable, efficient, and theoretically sound method for aligning models with diverse human preferences.

Strengths

1. Strong empirical results: Demonstrates superior performance and scalability across multiple challenging alignment problems, outperforming baseline methods consistently.· 2. Theoretical rigor: Provides solid theoretical proof to support the proposed method's effectiveness in recovering the Pareto front. 3. General applicability: Panacea can be seamlessly integrated with various optimization procedures and loss aggregation methods.

Weaknesses

1. Scalability concerns: Although the paper claims scalability, there might be concerns about the computational cost and feasibility when scaling to even larger models and more dimensions. 2. Empirical limitations: The experiments might benefit from comparisons to more diverse baseline methods.

Questions

1. Training costs: Include the training costs for reference. 2. Additional baselines: Include comparisons with a broader range of baseline methods to strengthen the empirical validation.

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

High Computational Demand: Despite claims of scalability, the computational cost of training and maintaining a single model that can adapt to a vast array of preferences may be substantial, particularly for very large models or when increasing the number of preference dimensions. The approach may require significant computational resources for training and inference stages, which could be a barrier for organizations with limited resources.

Summary

This paper addresses the Multi-Dimensional Preference Optimization (MDPO) problem, which involves aligning multiple objectives that exhibit heterogeneity in preference in the population, such as the tradeoff between harmlessness and helpfulness. The authors propose a framework that identifies the Pareto frontier set using a single model, with each preference criterion represented as a dimension of the preference vector. Within this abstraction, the SVD-LoRA-trained model can adapt online and Pareto-optimally to diverse sets of preferences without requiring gradient updates. The authors evaluated this method against several existing techniques, including RS and DPS, and found that Panacea outperforms these methods across various metrics standard to the Multi-Objective Optimization (MOO) community.

Strengths

1. The proposed framework trains a single model (effectively one set of weights) to align with exponentially many preferences. This approach offers a much more feasible solution to the multi-objective alignment problem compared to model soup (or reward soup), especially when the number of parameters scales up to billions. 2. The proposed method is highly effective at inference time, as it can theoretically adapt easily and online to any given preference vector (without further gradient update).

Weaknesses

1. The paper claims that the idea is theoretically sound. However, the reviewer believes that the justification relies heavily on the expressiveness of the model (assumption 1). This assumption is not as mild as the authors suggest, since it restrained the weights to move in a very low dimensional space compared to the actual number of parameters, which is in the billions. Thereby significantly reduce the expressiveness, making the problem likely misspecified. 2. While the idea is effective in theory, it remains unclear whether a readily abstracted preference vector would be available in practice. 3. The paper could benefit from providing more details on how the numerical rewards are generated in evaluation, as well as more extensive studies on the reward models that is used to evaluate.

Questions

1. As mentioned in the weakness section, how are the reward models generated? And most importantly how reliable are these models? 2. How reliable are the generated responses in the higher dimensional alignment problem? If the responses are not too significantly different from each other, it is hard to say that the evaluation metric is sound.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations in appendix.

Summary

This paper introduces Panacea for LLM alignment that reframes it as a multi-dimensional preference optimization problem. Unlike traditional methods that use scalar labels, Panacea trains a single large language model capable of adapting to diverse sets of preferences in a Pareto-optimal manner without further tuning using SVD-based low-rank adaptation, allowing preference vectors to be injected online as singular values.

Strengths

- This paper conducts thorough evaluation of the Pareto fronts obtained using metrics such as hypervolume, spacing, sparsity, and visualization for lower dimensional tasks. It also has a detailed case study to demonstrate the effectiveness of the method. - The method offers good scalability and scales up to ten dimensions. - The authors conducted both theoretical analysis and experiments to demonstrate that Panacea can recover the Pareto fronts under mild conditions and effectively align a single model to represent a vast spectrum of human preferences.

Weaknesses

- The presentation of figure 1 is a bit confusing and hard to understand. It can be improved. Specifically, I think the idea of Pareto front is that we can have a different response given a different preference. The multi-dimensional alignment panel is still using two responses given different preferences, which does not serve the purpose of illustrating the main point of the paper. - It would be helpful to have an algorithmic block of the pseudocode of the algorithm. - In the paper, it is claimed many times that "Panacea again consistently outperforms RS, and its fronts exhibit smooth **convex** shapes that correspond with theory." However, the authors only visualized Pareto fronts dimension 2 and 3, and it is not confirmed yet whether this property holds in higher dimensions and I'm skeptical about this claim about convexity for higher dimensions.

Questions

- The paper evaluates alignment up to tens of dimensions, including being humorous, philosophical, etc. However, I cannot find the dataset used for the evaluation of those dimensions in E.6 Information of assets Data section. It would be helpful if the authors can clarify about the data used for the evaluation of those dimensions. - Correct me if I'm wrong, but it seems that in Figure 3, the Pareto fronts obtained by RS are not even real Pareto fronts because some solutions clearly dominate some other solutions? If that's true, maybe it's worth commenting on in the paper.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors discussed the limitations as follows - Curse of dimensionality as dimension scales up to more than 10 - The method requires a preference vector from the user, but the user might not know a specific preference vector until they try a bunch of evaluations - No ground truth Pareto front available for evaluation

Nice rebuttal!

Thank you for your thorough and convincing rebuttal. I noticed that you evaluated your methods on 7B models. I’m curious about their effectiveness on smaller models as well. Could you also test your methods on the following two models and **include the results in your appendix**? Phi-3: https://arxiv.org/abs/2404.14219 MiniCPM: https://arxiv.org/abs/2404.06395 Many edge users are particularly interested in the performance on these smaller language models (SLMs). I apologize for the late response. Please let me know if there’s anything else I can do to assist.

Thanks for your appreciation

Dear Reviewer Chm5, Thank you for your positive feedback and for supporting our work. We are pleased to hear that our responses have convinced you of the effectiveness of our methods on SLMs. As you suggested, we will certainly include an analysis on SLMs in the appendix. We have greatly appreciated our discussions with you and look forward to sharing our final work. Best regards, The Authors