MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models

Summary

Multi-objective alignment of language models is a significant topic for LLM community, while many prior work are either costly-to-compute or policy-dependent, restricting the further depolyment. This work is aimed at providing a retraining-free and policy-agnostic approach for multi-objective alignment, namely *MetaAligner*. The framework is composed of 3 stages: - Data reorganization from multi-objective datasets, getting $D_e, D_p$; - Supervised model training: - Warming up on a randomly downsampled dataset of $D_e$; - Finetune on $D_e$, as a equal-preference alignment step; - Contrastive training on $D_p$. - Inference with prompting for even unseen objectives. Experiments are conducted on three commonly-used datasets, and the improvements are reported as prominent. Furthermore, the authors claim that they can achieve generalizable alignment to unseen objectives, through prompting.

Strengths

Originality: - The whole 3-stage framework of *MetaAligner* is original. Clarity: - This paper is well-written and well-organized. - Most figures and tables are friendly to the readers. Significance: - This is the first policy-agnostic and retraining-free multi-objective alignment approach, to the knowledge of the reviewer. - The experiments are extensive and very solid, including multiple datasets and base models. The author also conducted extensive experiments comparing *Meta-Aligner* with other baselines.

Weaknesses

*Major* - The whole framework is complicated and a bit engineering. There is no ablation study on each components. For example, it is better to show the performance if discarding the $D_e$ dataset, which is claimed as novel. - In Table3, the comparison of *MetaAligner* with other baselines is somehow unfair, since *MetaAligner* utilizes at least double parameters of *MORLHF*, *MODPO*, and *RiC* (the 1.1B version performs worse than baselines, so the reviewer is talking about >=7B version). - The proposed method is not able to cater to different users' taste, because it can only be aligned to multiple objectives while no weights on them are introduced. But many works[1][2][3][4][5] are able to do this, making the contribution less significant. *Minor* - this work is a bit marginal: - The reorganization process is just preparing the prompting format, which is trivial. - The SFT training is a common practice in the community[1][2][3]. The only difference is the design of a detached aligner. - The inference stage is simply prompting engineering, lacking guarantees. - The *MetaAligner* is not a light-weight approach, as the aligner model is still large, making it hard to be deployed. [1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402.10207. [2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. arXiv preprint arXiv:2402.19085. [3] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. arXiv preprint arXiv:2402.18571. [4] Decoding-Time Language Model Alignment with Multiple Objectives. arXiv preprint arXiv:2406.18853 [5] Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv preprint arXiv:2310.11564.

Questions

- In Figure 3, why would the performance on some unseen objectives drop after adding more objectives? For example, "repeat" and "readable" for *MetaAligner-7B*. - How much efforts has the author spent on tuning hyper-parameters of *MetaAligner*? - How does the author set the preference weights for *MORLHF*, *MODPO*, and *RiC*? Is that fair?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are well discussed in Section A.2.

Summary

The author proposes Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment.

Strengths

As a lightweight algorithm, this is particularly meaningful as model parameters continue to grow. Conditional weak-to-strong correction extends weak-to-strong correction and weak-to-strong generalization, which are currently significant research topics.

Weaknesses

1. I am not entirely clear about the difference between this paper and Aligner. The author mentions: > As shown, conditional weak-to-strong correction of MetaAligner extends Aligner [12] to multi-objective alignment scenarios, which are not directly solvable by Aligner itself. I do not fully understand the distinction between this paper and Aligner. After reading the Aligner paper, I noticed that the authors effectively expand on multi-objective alignment. However, the introduction seems to lack a description of Aligner, making it difficult for readers to understand. 2. I did not understand what the author is trying to convey with Figure 2. Can the author explain the meaning of Figure 2? The elements it aims to express are very complex. Could there be a more streamlined figure, similar to Figure 1 in the Aligner paper? Is the extension to multi-objective alignment at the methodological level or the dataset level? 3. In the experimental section, how did the authors generate an effective correction dataset based on HH-RLHF, UltraFeedback, and IMHI? 4. After reading the Aligner paper, I observed that it conducted experiments on the three dimensions of "Helpful," "Harmless," and "Honest" as described by the authors in line 42. I hope the authors can add comparative experiments with Aligner in the ablation study section. From a reviewer's perspective, the contribution of this paper to LLM alignment is undeniable, but I am particularly concerned about the effective improvement and extension of MetaAligner compared to Aligner. My questions are all included in the Weaknesses section, with point 4 being my primary concern. I hope the authors can effectively enhance the ablation study section. I believe that algorithms like MetaAligner and Aligner are more efficient compared to RLHF and DPO. I look forward to the authors' response and will consider raising the score further.

Questions

see above.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A.

Summary

This work proposes MetaAligner, a plug-and-play multi-objective alignment method that can generalize to unseen objectives.

Strengths

1. This is a well-written paper that clearly expresses its core ideas. 2. MetaAligner is a lightweight alignment method that is easier to tune in multi-objective scenarios, with lower training overhead compared to baselines. 3. MetaAligner is a plug-and-play alignment method that can generalize to various open-source and closed-source models, enabling multi-objective alignment. 4. MetaAligner has generalizability, capable of aligning with zero-shot objectives.

Weaknesses

1. In the experimental section, the aligned answers are compared to the ground truth answers in the dataset to calculate the win rate. This lacks a more direct performance comparison, making the argument for performance advantage insufficiently robust. For example, comparing different baselines by win rate against the ground truth answers in the dataset is inadequate to demonstrate the performance advantage over the baselines. 2. There is a lack of performance comparison with prompt engineering-based aligners. Specifically, it is unclear whether similar effects could be achieved by prompting the chat model to modify the answers within the prompts.

Questions

1. I am interested in understanding the authors' rationale for choosing an indirect comparison with the ground truth and how they view the validity of this choice, especially considering that many models, after being aligned through RLHF, are likely to already exhibit strong performance on the objectives of test set. 2. In the scenario where columns contain exactly two of the three attributes: Harmless, Helpful, and Humorous, which corresponds to the position of the second column in Figure 2 HH-RLHF. Let $( x_1, x_2, x_3 )$ denote the probabilities of having two characteristics, specifically including $Harmless$, $Helpful$, and $Humorous$. According to the figure, $( x_1 = 0.57 )$, $( x_2 = 0.53 )$, and $( x_3 = 0.85 )$. Now, there are exactly three possible combinations of choosing two out of the three characteristics: $(Harmless, Helpful)$, $(Helpful, Humorous)$, and $(Harmless, Humorous)$, and these combinations are mutually exclusive. Let the probabilities of these three events be $( a_1, a_2, a_3 )$ respectively, such that $( a_1 + a_2 + a_3 = 1 )$. The event `contains two characteristics, with one of them being Harmless` can be described as the disjoint union of the events `only contains (Harmless, Helpful)` and `only contains (Harmless, Humorous)`. Therefore, $( x_1 = a_1 + a_3 )$. Similarly, we have $( x_2 = a_1 + a_2 )$ and $( x_3 = a_2 + a_3 )$. Summing both sides of these three equations, we obtain $( 1.95 = 0.57 + 0.53 + 0.85 = x_1 + x_2 + x_3 = 2(a_1 + a_2 + a_3) = 2 \times 1 = 2 )$. I am sincerely inquiring whether my understanding of Figure 2 is mistaken.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The article provides a thorough discussion of its limitations in terms of deployment and the number of objectives to be selected.

Summary

This work extends Aligner to multi-objective alignment scenarios. The main contribution is adding a textual description to each sample in the existing preference datasets to indicate the reason for the preference between chosen and rejected samples. The authors found that MetaAligner is more efficient than previously trained multi-objective alignment algorithms. However, this work is incremental, with some unclear details, insufficient evaluation benchmarks, and a lack of baselines.

Strengths

Combining meta-learning concepts with alignment and guiding the model's alignment direction through textual descriptions makes sense to me and could potentially become an important part of RMs in future RLHF works.

Weaknesses

1. The innovation is incremental, essentially applying Aligner to a new domain and altering training data to change the alignment objectives. 2. The most critical detail, which is how the preference descriptions between each chosen and rejected pair in the existing preference datasets are obtained, is unclear. 3. The authors did not adequately present results on objective datasets such as MATH and HumanEval, nor on mainstream subjective evaluation sets like MT-Bench/AlpacaEval. 4. There is a lack of comparative analysis with previous multi-objective alignment methods, such as "SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling."

Questions

1. How are the objectives in Section 3.1 obtained?

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A