Group Robust Preference Optimization in Reward-free RLHF

Summary

This paper proposes a novel preference optimization technique, GRPO, which utilizes the group distributional robust optimization. Specifically, the method aims to maximize the worst-case group performance for improving the robustness of LLM models. This paper provides several theoretical aspects of GRPO when it comes to convergence guarantee and closed-form solution when considering the log-linear policy class. Empirically, the GRPO objective function is minimized by the mirror gradient descent. Several theoretical results on synthetic and real-world datasets demonstrate that their method works well.

Strengths

* This is the first study to adopt the group DRO technique in the RLHF setting. * Several theoretical results are provided to enhance understanding of the proposed method. * The proposed method is general so that it can be applied to various RLHF optimization techniques.

Weaknesses

* This paper lacks novelty in its contribution. The group DRO technique is already a popular method used in a variety of applications [1, 2]. While the authors offer some theoretical results and an algorithm for their method, these may also lack novelty. In specific, Proposition 3.1 on the Nash equilibrium is a well-known result in game theory and Proposition 3.2 seems straightforward because $\pi^*$ is dependent only on an input value, not on the group to which the input belongs. Furthermore, there is no significant difference in the algorithm and proof of convergence guarantee compared to existing works [37, 30 in the manuscript], except for slight modifications. * Although this paper is novel in applying the group DRO technique to RLHF preference optimization for the first time, stronger motivations and empirical results are needed to highlight the necessity of the proposed method. But, I also think that to strengthen this contribution, For example, as mentioned in lines 141-153, it would be beneficial to show that existing Large Language Models (LLMs) suffer from group robustness issues in various applications involving helpful/harmful instances or domain-specific categories when fine-tuned by RLHF optimization techniques. * The interpretation of the results in Section 5.2 is insufficient. In Figure 3, GR-IPO improved the performance of all groups, including Group 1. This result contrasts with findings reported in the Group DRO paper [37], where a trade-off between majority and minority group performance is generally observed, with majority group performance typically decreasing to compensate for minority group performance. Therefore, a more thorough analysis or performance comparison is needed to explain this result. [1] Re-weighting based group fairness regularization via classwise robust optimization, 2023. [2] Distributionally Robust Multilingual Machine Translation, 2021.

Questions

1. In the experiments in Section 5.2, what are the zero-shot performances of Gemma-2B for each group? It would be better to report the degree of bias in the pre-trained model together, because it can enable comparing the bias degrees between the pre-trained and fine-tuned models. 2. Would you provide the results when fine-tuning more layers or the entire neural networks using your method? I think that this can strengthen this paper's contribution by showing the performance of the method in various settings.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

They addressed the limitations of this paper in the manuscript.

Summary

This paper addresses the limitation of traditional reinforcement learning with human feedback (RLHF) approaches that indiscriminately optimize a single preference model, disregarding the unique characteristics and needs of diverse labeler groups. The authors propose a Group Robust Preference Optimization (GRPO) method to align LLMs with individual groups' preferences.

Strengths

The paper's motivation is clear.

Weaknesses

1. How does the proposed method handle scenarios where preference data lacks clear group information? In practice, much preference data does not come with explicit group classifications. 2. The paper should address whether the proposed method can learn an invariant preference standard from data that may contain various group information, such as different preference evaluation criteria and preferences annotated at different iterations. 3. The authors need to provide and analyze the alignment results of the model, not just the reward model analysis.

Questions

Please refer to the Weaknesses section for questions regarding the paper.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

The work tackles the important problem of robust RLHF for diverse groups. Traditionally, RLHF assumes that a single model can fit the diverse feedback from multiple groups of users. In this paper, the authors introduce a method to learn a robust policy that maximizes for worst-case group performance. To achieve this, the method adaptively weights the loss for each group based on the size and cumulative training loss incurred by the feedback samples for that group. As a result, LLMs trained on diverse group data demonstrate reduced loss imbalance and improved accuracies across all the groups. The authors also present a convergence analysis of the proposed method assuming a log-linear policy class.

Strengths

1. The paper tackles a critical problem of robust RLHF for diverse groups. The intuition of the overall method is well understood, the framework and the parameters are clearly mentioned, and the results are shown over the standard datasets and compared to multiple baselines. 2. The authors present a thorough literature review and background. Past work has mainly focused on making RLHF and LLMs robust to noisy or out of distribution data. Meanwhile, this work focuses on a group robust formulation of training LLMs using state-of-the-art methods (mainly DPO). 3. The method introduced “GRPO” is useful is scenarios beyond diverse groups. As the authors mention, it is a general formulation that can enforce robustness to diverse tasks, domains, or objectives occurring in the feedback dataset. 4. To achieve robustness, GRPO presents a robust optimization approach to minimize worst-case loss amongst the diverse groups. Further, the paper introduces a less aggressive object by trading off worst and average case performance. I would be curious to see an ablation study showing the effectiveness of this tradeoff. 5. The paper supplements the approach with some strong theoretical proofs on the convergence properties of the method under a log-linear policy. The authors also present a close form solution for the RLHF update step, replacing DPO with IPO. 6. The results show that GRPO improved performance across all groups, and the weight update behaves as expected by assigning higher weights to groups with higher cumulative loss i.e. the gradients from the worst performing group are scaled the most.

Weaknesses

1. This paper proposes a method for group robust optimization for LLMs. However, the metrics evaluated are only max validation loss and reward error over the groups. GRPO uses a reward free approach to update the LLM, but the evaluations are restricted to the performance of the reward model over the feedback dataset. It would be nice to see the performance of the finetuned vs base model (such as win rate) in generating responses that align with the individual groups. 2. The authors provide detailed training setup, but I would suggest that they also include information about the evaluation. 3. GRPO performs well over all the groups, however, the performance of the importance sampling baseline is very close. It would be helpful if the authors could provide additional ablations and discussions to show the effectiveness of optimizing against the worst-case loss over the groups vs using only the IS approach.

Questions

1. In the current experiments, is only the final layer of the LLM trained? I would be curious to know the result if the method used full finetuning of the model. 2. As GRPO assumes that each prompt has access to group information through a prompt, how much does the prompt affect policy? If the prompts are good enough, would it just create non-overlapping distributions for all groups? As one of the problems in non-robust baselines is that they converge to unwanted biases or majority groups for shared prompts, if the group identification prompt is finetuned will it alleviate this issue altogether? 3. Does GRPO finetune only the final layer of the Gemma-2B model in the results presented in the paper? 4. Does GRPO ensure that there is a non-decreasing change in performance across all groups as compared to the non-robust baselines?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

1. The paper introduces GRPO, an optimization method to weight each group in the RLHF update step proportional to the size and loss of the group. This ensures a balanced performance of the model across all the groups. In the results, we see an improved performance over all the individual groups, which intuitively violates the no-free lunch theorem. I would be curious to see an analysis of the method that gives insights as to what allows the model or the objective to achieve consistently higher performance. 2. Here, the method assumes access the groups in the dataset. However, in practical settings the group information is unavailable and the model has to cluster or implicitly model the group information from the ungrouped dataset. 3. The evaluation is limited only to the accuracy of the reward model over the preference dataset. So, currently, it provides weaker evidence of the translation of this robustness during the generation phase. It would be nice if the authors could include experiments showing if GRPO enables LLMs to robustly generate better-aligned responses to prompts from all the groups.

Summary

This paper introduces GRPO, a method to optimize policy preferences across different groups in a robust way. GRPO looks at the worst group alignment loss by taking the maximum loss across all groups, ensuring the policy performs well even when there are group-specific differences or overlaps in prompts.

Strengths

1. This paper writes well and easy to follow. 2. The paper provides a thorough theoretical analysis of GRPO. 3. The proposed framework accommodates both distinct and overlapping group scenarios. 4. GRPO can trade off worst-case for average performance with a hyperparameter.

Weaknesses

1. GRPO's experiment on real world datasets is limited, on only one group (5 countries) of the GlobalQA dataset. 2. How is the training time as compared to other baselines? The GRPO framework's training process involves a min-max optimization, which can be potentially computationally intensive.

Questions

please see weakness.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

1. This methods requires knowing the number and nature of groups in advance to perform optimization. 2. When new group is introduced, will have to train the entire optimization again, which is expensive.

Summary

This paper addresses the problem of improving fairness in preference optimization. It proposes a new loss function and algorithm. Experiments conducted show that the proposed algorithm indeed achieves better fairness. Additionally, the paper provides a theoretical analysis indicating the convergence of the optimization.

Strengths

1. The idea of improving the fairness is good. This is a problem that deserves more attention. 2. The derivation of the loss and the algorithm, though simple, is clear. 3. The results provided for the algorithm appear to be sound.

Weaknesses

1. The author mentioned that the idea can also be applied to multi-objective preference optimization. In my opinion, a comparison with works in multi-objective preference optimization [1] and group optimization [2] is needed. 2. It is unclear whether this algorithm can be applied to scenarios beyond multiple choices. 3. It is unclear how Proposition 4.1 relates to common metrics like regret. [1]. Zhao, Siyan, John Dang, and Aditya Grover. "Group preference optimization: Few-shot alignment of large language models." [2]. Wang, Haoxiang, et al. "Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards."

Questions

1. The paper mentioned that the algorithms can also be used for multi-objective preference optimization. Were any related experiments conducted?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

See the 'Weakness' part

Response to Ethics Reviewer ipoT

We thank the reviewer for their review and recommendations. In our humble opinions, our work is proposed to address the potential ethical issues in current LLM training, i.e., our robust alignment can ensure more equitable LLMs across different demographic groups. Regarding your questions, which are raised by other reviewers and addressed, we reiterate the answer. 1) The popular datasets used in pluralistic alignment research typically have a well-defined group notion. Our formal problem statement explicitly mentioned that we consider the case of known groups. Moreover, our approach is independent of whether the groups are mutually exclusive. We treat them as distinct identifiers/contexts (and not as sets that can overlap) and we are optimizing the worst-case. When the group information is unknown, one may apply methods to identify unknown groups. a) Clustering: Apply algorithms like k-means or EM to detect potential groups; b) Representation learning: Use techniques like variational autoencoders (VAEs) to discover hidden structures; c) Analyzing features: Look for features with high variance or correlations to hypothesize groupings; d) Consulting experts: Leverage domain expertise to identify meaningful groups. These methods can help uncover group structures in the data. We will include the above paragraph in the manuscript and acknowledge that this is an interesting avenue for future research. 2) We clarify that we do not claim that our method will improve performance across all groups over the non-robust baseline. In general, minimax methods improve the worst group’s performance, often reducing the average or best group’s performance (“no free lunch” property). Our proposed GRPO formulation (Eq. [8]), stemming from a minimax framework, will comply with the same property. We will make it clear in the next version. The observation of improved performance across all groups in one of our plots is discussed in further detail in our response to reviewer kmNK. We kindly refer the ethics reviewer to that. Regarding the tradeoff over worst-case vs average-case performance, we discuss in detail in Appendix B.4, wherein we introduce a trade-off parameter $\chi$ to balance the worst-case vs average performance. In response to reviewer kmNK, we ran ablation experiments on this trade-off parameter to compare performances at different levels of importance to worst-case vs average performance and plot them in Figure R1 of the attached PDF. Note that the max validation loss decreases while moving from $\chi=0$ to $\chi=1$, where $\chi=0$ corresponds to importance sampling (see Eq. [30] in Appendix B.4) and $\chi=1$ corresponds to GR-IPO. Further, we plot the average validation loss which increases while moving from $\chi=0$ to $\chi=1$, demonstrating the trade-off between average and worst-case performance. Note that, GR-IPO aptly increases the average loss (as expected) in order to reduce the worst group loss. These discussions will be included in the next version.

Response to Ethics Reviewer CLC3

We thank the reviewer for their review and recommendations and for pointing out that our manuscript **contains no ethical issues.** We provide the answers below: - We strongly disagree with the reviewer that our statement is not supported by evidence. As explored in both the synthetic and real-world experiments, due to inherent structures in the data and the pre-trained model’s capabilities, there is a significant bias in the performance of non-robust baselines across groups. Our method improves the worst group performance, thereby mitigating the biases in alignment performance across groups. This is further evidenced in Figure R3 of the attached PDF, where Group 5’s performance improves albeit at the expense of Group 1, hence ensuring a more equitable alignment. We also want to emphasize that we were very careful in our phrasing, which is why we used cautious language, such as "hold promise”. - We note that “distinct” does not imply “non-overlapping”. We believe that even different overlapping groups can be considered distinct. If the reviewer still finds this confusing, we are happy to reword the sentence. Our approach is independent of whether the groups are mutually exclusive. We treat them through distinct identifiers/contexts, rather than overlapping sets, and optimize for the worst-case performance across groups. While some groups may have preference distributions that are more “similar” to others, others may have “fairly” different preference distributions. We observed this in our real-world data experiment in Section 5.2. However, this has no impact on the performance of our method.

Gentle Reminder

We thank all the reviewers again for the time taken to review our work. As the author-reviewer discussion period is nearing its end, we would appreciate it if the reviewers would respond to our rebuttals. If there any further questions, we are more than happy to clarify and discuss further.