Axioms for AI Alignment from Human Feedback

Summary

Authors argue that for RLHF, the preferences are pairwise and we need to train a model that respects the preferences in aggregate which is in scope of social choice theory. Then they evaluate different aggregation methods on if they respect well established axioms. They showed that the popular TB model does not respect some axioms and came up with novel rules for reward learning.

Strengths

- principled approach to model preferences from population, rooted in social choice theory - relevant topic for this conference - initated the filed to approach RLHF reward modeling to follow axiomatic guidance through social choice theory

Weaknesses

- only theoretical contributions, we do not know how the theory translates to in practice, even in toy settings

Questions

- could you cite the places where you obtained the axioms of social choice theory? - since you did not include where these axioms are from, are you missing any investigations of other axioms? If so, why did you choose the ones you investigated and why leaved the others out?

Rating

7

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

written in the discussion section

Summary

Recent months have seen a flood of concurrent papers studying the relationship between RLHF, preference aggregation, and social choice theory. This paper joins these lines of work and studies how to aggregate diverse human preferences (in the context of RLHF) that are modeled as a random utility model (e..g., BTL). The authors adopt a social choice perspective and show that the BTL model (and similar ones) fail to satisfy basic axioms known from social choice theory. The paper then also proposes a leximax Copeland (subject to PO) rule that satisfies desirable properties such as pareto optimality and majority consistency.

Strengths

- The paper is very well-written. - The studied problem is relevant and timely. - The authors do a good job explaining concepts from social choice theory so that the paper is also easy to read for researchers who don't have a background in computational social choice. - While unsurprising, the main theoretical result that linear aggregation (using a linearly parameterized reward function) is insufficient to achieve basic axioms of "fair" and reasonable preference aggregation is useful and interesting.

Weaknesses

- The assumption that for each voter the complete ranking is available is obviously unrealistic, which limits the applicability of the proposed leximax Copeland rule. - I believe that this paper's practical relevance is quite limited. It is unclear how leximax Copeland could be implemented in practice. The sampling strategy over voters and candidates (prompt/responses) is far from obvious and not discussed in the paper. Any additional comments about how such a sampling strategy would look like would be appreciated. - In contrast to the various related works on RLHF + social choice, which provide algorithms/solutions to augment the traditional RLHF framework, this work does not provide a tractable approach to address the problem of aggregating preferences in a reasonable way.

Questions

- What about non-linear reward function classes?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors address the limitations of the work adequately in Section 5.

Summary

The paper presents a axiomatic social choice framework for the problem of doing RLHF on group preferences. It shows that classical RLHF (and indeed a wider class of similar methods) violates the Pareto Optimality and Pairwise Majority Consistency axioms, and shows (via explicit construction) that there are mechanisms that satisfy both axioms along with two other axioms. (To simplify the model, the space of preferences is assumed to be implementable via linear classification.)

Strengths

**High importance and novelty** - LLMs, and also AI systems in general, are expected to have large societal impact now and in the future. In light of this expectation, how to make sure model policies fairly represent the welfare of all stakeholders is important. This question, to my knowledge, has not received sufficient analysis except in this work and some of its concurrent works. **Flawless construction of the theoretical framework** - The theoretical setup seems flawless, covering exactly all the fundamental components of the problem, and in a very elegant manner.

Weaknesses

I have some suggestions for improvement/future work, but I don't feel like they count as "weaknesses" *per se*, since those are very high standards which I don't feel like a normal conference paper would be held up to. I am instead putting those suggestions in the limitations section. Also, please note that I did not check the proofs.

Questions

- Do you think the PO and PMC axioms (possibly along with majority consistency, winner monotonicity etc.) are in any sense the "gold standards"? Could there be other similarly reasonable axioms that are contradictory with PO/PMC? I'm asking this because I have the intuition (which could be wrong) that an Arrow-like impossibility result would also haunt the linear social choice setting; elaboration in the limitations section.

Rating

8

Confidence

3

Soundness

4

Presentation

3

Contribution

4

Limitations

I highly appreciate the detailed discussion of limitations and future directions in the Discussion section of the paper, and I agree with most of the points there. I have the following two additional remarks, which are meant not as critiques but as suggestions for exploration. 1. **(Non)existence of a gold standard in linear social choice** - Let's look at Theorem 3.1 first. If the RLHF method decide to overturn a perfect consensus on one pair with a tiny margin, in order to much more significantly reduce loss on a divisive pair (which is possible due to strict convexity), this actually seems desirable, despite violating PO. - One could counter that whether the margin is small or large does not matter when what we care about the outcome is only the ordering. However, the exact margins (<> the exact reward values) do matter a lot in practice, where they indirectly (in RLHF) or directly (in DPO) determines what probabilities to assign to each response (as opposed to merely a ranking relation between the responses). - In general, I suspect there is some arrow-like impossibility result in this space, where all the desirable properties just cannot be met at the same time. It's unclear which among the conflicting properties (somewhere among them are PO and PMC, and somewhere else is "prioritize preference violations with larger margins", with many others) are the most important. - This intuition seems to be confirmed by the LCPO construction, which introduces rather arbitrary requirements (namely hard-coding the PO rule into the LCPO algorithm) in order to satisfy PO. 2. **Human evaluations and human subject experiments** - I suspect that human evaluation (e.g. letting human subjects judge the fairness of preference aggregation outcomes) could be an equally, if not more, important criteria for preference aggregation mechanisms than formal axioms are, given that (1) it's unclear which formal principles are more important than others and (2) these principles could be in conflict with each other (as in Arrow's theorem). - This could be used to evaluate both aggregation mechanisms (does the outcomes align with human judgment?) and axioms (is human judgment in line with this axiom in general? If not, why, and who is right?). - Here, the line between computer science and cognitive science seems to be dissolving.

Summary

The paper proposes an axiomatic approach to study preference aggregation for AI alignment. Inspired by works in social choice theory, the authors investigate a paradigm that they call linear social choice where preferences are representable by a linear model. In this context, they notably prove that if the linear rank aggregation rule minimizes some natural loss function (like the one induced by the Bradley-Terry model), then it cannot satisfy Pareto optimality. To circumvent this issue, the authors propose a variation of Copeland rule that outputs a ranking representable by a linear model.

Strengths

The paper investigates an important research question with a theoretically-founded approach. The obtained results seem to be novel as far as I know. The failure of PO by minimizing a loss points to a strong limitation of the current approaches using human feedback. The paper is well and clearly written. I notably appreciate the footnotes, comments, and connections with other works that the authors make.

Weaknesses

I feel that the linear social choice paradigm may be too restrictive. I believe that the authors focus on linear models, because this is indeed a natural machine learning model. However, I don't think it is necessary to assume that the rankings of voters (i.e., humans in AI alignment) have to be linearly representable. In my opinion, the paper would be stronger if the results were presented without this latter assumption. There are some gaps between the setting in machine learning (e.g., RLHF) and social choice theory. As the authors mention, in the latter, one usually observes pairwise comparisons instead of rankings. In addition, I think in the latter, one may not be interested to recover the full ranking, but only part of it, e.g., in RLHF, as the agent is trained, the reward approximator only needs to be good in the region of state-action pairs visited by a good policy. Could the authors comment on how this would impact their results? The proposed social choice-based rule seems to me to be a bit artificial and only created to enforce Pareto optimality. It is not clear if such rule could easily be implemented in RLHF for instance and obtained via learning. For instance, in RLHF, this rule would require to consider all trajectories, which is impractical for any complex problem.

Questions

1) Could the authors comment on how only trying to recover part of the full ranking would impact their results? 2) Could you the authors comment on their proposed rule could be implemented in practice?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

2

Limitations

The current discussion is adequate.

Reviewer - Author Discussion

Thanks everyone for their hard work on the papers, reviews, and rebuttals. We now have a comprehensive rebuttal from the authors which responds both overall and to each review. I'd please ask the reviewers to please post a comment acknowledging that they have read the response and ask any followup questions (if any). Thanks to those of you who have already done this! This period is to be a discussion between authors and reviewers (Aug 7 - Aug 13) so please do engage now, early in the window, so there is time for a back and forth. Thanks!