Aligning to Thousands of Preferences via System Message Generalization

Summary

In this paper, the authors introduce a novel method to align LLMs with diverse user preferences without requiring continual retraining for each specific preference. The approach utilizes a unique system message protocol that guides LLMs to produce responses tailored to specific, nuanced user preferences.

Strengths

1. The idea is novel: adapting to different users' preferences by training on preference data with different system prompts. 2. It is intuitively a useful research, effectively strengthening the alignment effectiveness by clarifying the user identities besides the alignment data themselves. 3. The paper is well-written and easy to follow.

Weaknesses

1. How many humans did you hired to perform the human evaluation? There lacks a illustration on this. If we want to illustrate its ability to cater to a wide range of peoples with different backgrounds, then we should also apply a wide range of people to evaluate. 2. I feel that there lacks an illustration to show that the model is not becoming verbose. As all of the benchmarks you use prefer verbose responses.

Questions

Is this a way to "hack" the Chatbot Arena? (in the right way lol)

Rating

6

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

Yes

Summary

This paper addresses the issue that humans inherently have diverse values, while current LLMs are primarily aligned with general public preferences such as helpfulness and harmlessness. Previous work has trained new reward models (RMs) and LLMs for individual preferences, which is time-consuming and costly. The authors propose a new paradigm where users specify their values within system messages. Specifically, they create the Multifaceted Collection, containing 65k user instructions, each with 3 system messages, resulting in 192k combinations of fine-grained values with the assistance of GPT-4. They then train mistra-7b-v0.2 into Janus-7B and test it on 921 prompts collected from 5 benchmarks. Compared to leading open-sourced and proprietary models using pairwise ranking human evaluation or GPT-4-as-a-judge, Janus-7B achieves a higher win rate while maintaining harmlessness.

Strengths

The paper proposes a new paradigm to address the problem of aligning to fine-grained values without repeatedly training multiple RMs and LLMs. It also creates a large-scale preference dataset with fine-grained system prompts representing multiple alignment targets. Detailed experiments across many leading open-source and proprietary models validate the method's effectiveness.

Weaknesses

While the paper is well-written and presents a solid analysis, several weaknesses need addressing: 1. The construction of the Multifaceted Collection is costly and not scalable. GPT-4 generates preference sets for each of the 65k instructions, converts each of the 198k preference sets into a system message, and crafts gold-standard multifaceted responses for each system message. 2. The role of system message generalization has not been clarified. What are the differences between varying the system messages for individualized alignment targets and simply using different instructions in the user prompt while maintaining the system message unchanged to reach the specified alignment targets?

Questions

1. What are the differences between varying the system messages for individualized alignment targets and simply using different instructions in the user prompt while maintaining the system message unchanged to reach the specified alignment targets? It seems that I can just use specialized instructions in user prompts to reach the same goal of aligning to individualized values. Can you comment on this? 2. There are several typos: - Line 136: 5 datasets, not 4 - Line 152: Appendix B.1? - Line 286: Appendix G.1? And so on

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The construction of the Multifaceted Collection is costly and not scalable. And the role of system message generalization is unclear. It is not explained how varying system messages for individualized alignment targets differs from using different instructions in user prompts while keeping the system message unchanged.

Summary

This paper aims to align Large Language Models (LLMs) with individual user preferences at scale. The authors propose a paradigm where users specify their values within system messages to guide the LLM's generation behavior. To address the challenge of generalizing to diverse system messages, the authors create the MULTIFACETED COLLECTION, a dataset with 192k unique combinations of values and instructions. They train a 7B LLM called JANUS using this dataset and demonstrate its effectiveness in adhering to various user preferences without the need for retraining for each individual. The JANUS model outperforms other models like Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4 in benchmarks that assess response helpfulness.

Strengths

1. The creation of the MULTIFACETED COLLECTION, as well as the trained models, makes a good contribution to the community for studying diverse preference alignment. 2. The idea of utilizing system messages, while simple, is demonstrated as a scalable solution to the problem of individualized LLM alignment. 3. The paper shows that training with a diverse array of system messages not only supports personalized responses but also enhances alignment with general public preferences.

Weaknesses

While widely adopted, the GPT4-based data synthesis process may introduce some artifacts. The authors should include quantitative and qualitative analyses to investigate the potential biases and representativeness in human preferences in the collected MULTIFACETED COLLECTION.

Questions

How do you measure the diversity of user preferences in MULTIFACETED COLLECTION?

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations.

Summary

This paper introduces JANUS, a novel approach to aligning large language models (LLMs) with diverse individual user preferences without retraining. The key contributions are the MULTIFACETED COLLECTION: A dataset of 192k diverse system messages reflecting varied user preferences, paired with 65k instructions. JANUS: A 7B parameter LLM trained on this dataset to generalize to unseen system messages at test time. Evaluation showing JANUS outperforms several larger models, including GPT-3.5 and GPT-4, in adhering to specified preferences.

Strengths

1. Using diverse system messages for alignment is novel and creative. It tackles the challenge of personalized alignment in a scalable way, avoiding the need to retrain models for each user. 2. The methodology is rigorous, with careful dataset construction and comprehensive evaluations against strong baselines. The use of human evaluation alongside automated metrics strengthens the results. 3. This paper is well-structured and clearly written. The approach and results are explained thoroughly, with helpful figures illustrating key concepts. It work has potentially high impact, addressing a crucial challenge in AI alignment. The ability to adapt to individual preferences without retraining could be transformative for deploying personalized AI systems at scale.

Weaknesses

1. The paper doesn't deeply examine potential risks of allowing such flexible preference specification, such as potential for misuse or unintended biases. - There's no discussion on how to handle conflicting or ethically questionable preferences. 2. While the overall approach is effective, it's unclear which components contribute most to the performance gains. For example, how much does the hierarchical nature of the preference augmentation contribute versus simply having a large number of diverse preferences? The impact of the number of unique system messages or instructions is not explored. 3. The study focuses on 7B parameter models, primarily comparing to other 7B models or larger. It's unclear how well the approach scales to smaller models or much larger models (e.g., 70B+). The comparison to LLaMA 3 8B Instruct is promising, but more exploration of scale effects would be valuable.

Questions

1. Have you explored any potential negative consequences of allowing such flexible specification of preferences through system messages? Are there safeguards to prevent misuse or handling of conflicting ethical preferences? 2. How does the performance of JANUS change as you vary the number of unique system messages in the training data? Is there a point of diminishing returns, and how does this relate to the hierarchical structure of your preference augmentation? 3. Have you tested the approach on models significantly larger or smaller than 7B parameters? How does the effectiveness of this method scale with model size, especially compared to the scaling behavior of traditional fine-tuning approaches? 4. How consistent is JANUS in maintaining the specified preferences across a long conversation or multiple diverse tasks? Have you evaluated this aspect, and if so, what metrics did you use? 5. The paper mentions outperforming larger models like GPT-3.5 and GPT-4. Have you analyzed why your approach seems to be more effective than simply scaling up model size for this task? Could this indicate a fundamental advantage of your method over scale alone? How did you ensure the quality and diversity of the generated system messages and reference answers in the MULTIFACETED COLLECTION? Did you implement any specific measures to mitigate potential biases introduced by using GPT-4-Turbo-0125 for generation?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

None.

Looking forward for Discussions

Dear Reviewers, We appreciate for your time and effort on reviewing the paper. As we near the end of the discussion period, we kindly remind you of the upcoming deadline. We are eager to discuss any aspects of the paper that may require further clarification. Thank you once again for your valuable feedback.