Summary
This paper is concerned with the study of scalable oversight methods, i.e, how can one devise methods that will allow humans to supervise and align superintelligent models (ASI) whose capacities (which include reasoning, strategic thinking, and deception) vastly exceed the ones of humans. Inspired both by recent works studying debate as a method for aligning strongly capable AI and by works modelling scalable oversight with smaller LLMs tasked to align stronger LLMs, the current work studies debate between more capable LLMs, as judged by a weaker LLM, both as a proxy for scalable oversight of ASI by humans and as a proxy for the richness of the signal the judge could provide to the strong LLMs during alignment training.
The authors back the task on extractive and closed QA tasks with 2 possible answers, where each of the 2 debating models are given a side and must persuade the judge to agree with them. This debate task is contrasted with the consultancy task, where a single model is given an arbitrary side and must persuade the judge to agree with it (the arguments for the opposing side are not visible). The study is large-scale, involving 9 tasks totaling 128 questions. A set of LLMs of varying sizes are used as judge to assess the effects of the gap between judge and debater capabilities. Open variants of debate and consultancy, where the judge/consultants are allowed to choose the answer they will argue for, are also investigated.
Among the important results of the paper, the authors find that under debate, for all judge sizes, the judges achieve better accuracy (predicting the correct answer) than under consultancy, highlighting the promise of debate as a promising alternative to RLHF as a basis for scalable oversight. Then, then show that in their setting, judges are convinced equally by consultants that have chosen the right versus the wrong answer, whereas in the debate case models that have chosen the right answer are believed more often, providing additional evidence for debate vs consultancy. Then, they show that judge accuracy increases as the capabilities of debaters increases (measured by their Elo in debates against other models), showing that debate scales with the capabilities of the LLMs to align. Their result also extends previous work on the debate task, with judges of the same strength as debaters, that was performed in a single task.
Strengths
* The subject matter is important and of overarching importance to the Neurips community and beyond; the findings will be of particular interest to anyone concerned with AI safety.
* The paper is excellently written. The subject is not completely trivial and there are many setups and extensive experiments to present, but nevertheless the authors do an excellent job of explaining everything and putting it all together, highlighting the main results and the lessons learned as they go along as well as in the introduction and conclusion. The paper is very well contextualized in the related work and relations to prior art are precisely explained and motivate the current approach. I am not an expert in scalable oversight but I feel I have a much stronger grasp on the domain after reading the paper;
* The paper proposes to extend the study for a candidate for scalable oversight which follows naturally from previous work by casting it in a setting that is partially representative of the challenges ASI alignment poses. The authors study this as a scientific question, precisely reporting their findings and not overstating the extent to which debate is a definitive solution to scalable oversight. Extensive experiments support all of their claims and conclusions, and overall the paper (and its appendix) are information-rich.
* The authors highlight where their results agree or disagree with previous results in the literature.
* While debate as an alignment method is not novel it has not been studied with weaker judges judging stronger models, nor with such task variability (including knowledge-intensive tasks and reasoning-intensive tasks).
* All results come with clearly marked 95% CIs.
Weaknesses
* One could have hoped that debate would perform better, compared to consultancy, but the gap between methods is still small (while consistent). How could one improve on debate to create a stronger signal for alignment? (this is hardly a weakness of the paper, however, but potential solutions to this could be discussed in the paper)
* Reproducibility is not perfect, since some results make use of chatgpt;
Questions
* I like the result on chain of thought, it is counterintuitive and the explanation is plausible. Any idea on how to test this? (Maybe looking at attention matrices, or token influence?)
* line 275 I’m confused as to how models can both exhibit systematic positional bias and judge accuracy by unaffected by evaluation in both orders. How can this happen?
* line 344 “we don’t see such a clear trend of this advantage with increasing Elo” why do you think this is the case?
* (very minor) Summary sentences have too many commas, feels not that fluid (l353-357);
Limitations
The main limitations of the work have been addressed by the authors at length in their paper, as far as I can tell. The expected societal impact of the work is likely to be overwhelmingly positive, as is usually the case with safety research. Maybe one note is that all alignment research is potentially misalignment research, in the wrong hands -- but this is hardly specific to this paper.