Prediction-Powered Ranking of Large Language Models

Summary

The paper studies uncertainty estimate in the LLM ranking problem, where the task is to rank LLMs based on their response quality. Ideally the labels should be from humans, but due to the cose, people use models such as GPT-4 as auto raters. There lacks a good study of uncertainty estimation in the problem. The paper applies prediction powered inference (PPI) to construct a rank set for each candidate LLMs. Experiments are conducted on the Chatbot Arena data. It is shown that the PPI approach work as intended to produce reasonable rank set size vs accuracy (when comparing with an oracle method using only human data) trade-offs.

Strengths

The application of PPI to constructing rank set from pairwise comparisons from LLM evaluators is an interesting and timely application to the reviewer. The proposed algorithms look sound by following PPI. Overall the experiments demonstrate some desired behaviors of the proposed approach. Some analysis, such as the structure of the rank-sets, are interesting.

Weaknesses

Overall I am positive of the paper as I feel the problem is important and applying PPI is a good proposal. However, the reviewer is not enthusiastic enough to give a higher rating due to the following concerns: The paper is a more or less straightforward application of PPI to rank set construction. The theoretical properties shown in this paper mostly strictly follow those of PPI. Thus, the depth of this work is not substantial enough to warrant a higher rating in terms of novelty and technical depth. The rank set is at dataset level, which may not be very useful in practice. For example, compared to a standard usage of PPI on a numeric metric, which will provide a concrete internal. For the paper, people end up with a discrete rank set for each candidate model. While it provides some uncertainty by looking at the set size, one may still end up wondering how useful that is. For example, if model 1 has rank 2,3,4, and model 2 has rank 3,4,5. It provides some information that model 1 seems better, but the reviewer is not sure how useful it really is (e.g., by how much? - as the guarantee is at the set level). There are several points about the experimentation that the reviewer is not certain about - As the authors acknowledged, only one dataset is used so the generalization is less clear. F - The baseline / ground truth still needs some processing, such as a regression fit. This is different from standard tasks where human ground truth are given without the need to process anything. Thus the reviewer is not fully convinced how solid the conclusions are, e.g., “questioning the rationals used an extensive line of work that … (use LLM rankings for evaluation)” - Figure 1 is not entirely convincing - at least the PRP methods do not achieve pareto frontier here. The reviewer understands the argument about the x axis, still, for a pareto problem, one may not really argue one method is better than the other if it is not pareto optimal. The flow of the paper may be improved. The reviewer was puzzled about the methods and algorithm 1,2,3 when reaching to the experiments. Figure 1 is not very easy to interpret. Some other minor limitations / future work, some are discussed in the paper: iid assumption - in practice, there could be a bias, e.g. using active learning to send data to human.

Questions

See above.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors list several concrete limitations. Most are treated as future work.

Summary

This paper proposes a statistical framework to rank a collection of LLMs according to how well their output aligns with human preferences. This framework does this using a small set of human-obtained pairwise comparisons from LMSYS Chatbot Arena platform and a larger set of pairwise comparisons by a "strong" LLM and additionally provides an uncertainty estimate by giving a set of rankings for each LLM being compared. This study shows that, with at least probability threshold the user can set, the predicted rankings will eventually become increasingly likely to match the true order in which humans would prefer the models. The authors perform several experiments to empirically demonstrate the valitdity of the proposed framework.

Strengths

- The framework is clearly explained and the paper is easy to follow - The paper studies an interesting problem that focuses on ranking LLM in the context of scarcity of gathered pairwise comparisons by humans - The empirical evaluation is thorough

Weaknesses

- unless i misunderstood something, the small set of human pairwise comparisons has length = 1. Although, the potential bias and truthfulness of the human pairwise comparisons has been discussed in the limitation section, i think that it could be interesting to explore the potential error propagation in the rank sets from erroneous human comparisons. - it is not clear to me, how the self-recognition [1] problem can be tackled with this framework. It has been shown that LLMs have non trivial capability of recognizing their own generation. Would this not be the case for one of the strong LLMs? wouldn't they tend to rank their generation higher? This fact combined with the previous question might affect the generazalization ability of this framework. [1]: Panickssery, Arjun, Samuel R. Bowman, and Shi Feng. "Llm evaluators recognize and favor their own generations." arXiv preprint arXiv:2404.13076 (2024).

Questions

see weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

limitations have been discussed.

Summary

- Focuses on uncertainty in rankings using a small set of human pairwise comparisons and a large set of model estimated comparisons using a concept of rank sets. A rank set is a set of ranks a specific model can take. A large rank set indicates high uncertainty in ranking position and vice-versa a small set implies a confident rank assessment. The method works by constructing a confidence ellipsoid which in turn using methods of prediction powered inferences (methods to estimate confidence intervals when you have a small set of labeled gold standard data and a large set of machine labeled data). The paper employs the methods to rank 12 LLMs using data from LMSYS Chatbot Arena.

Strengths

- Evaluations of LLMs is particularly challenging and estimating ranking of models for specific tasks an important area. This paper makes a good contribution towards this by investigating ranking uncertainty using prediction powered inferences. The overall setting (small set of human-annotated data and a large machine labeled dataset) is realistic and therefore the work lends itself to practical use as well. - The methodological contributions are interesting in itself and the concept of using rank sets to characterize uncertainty intriguing. - The exposition and presentation of material is good. Some of the plots are well structured and intuitive to grasp (Figure 3 in particular is well crafted).

Weaknesses

- The main weakness is on evaluation. The paper proposes two metrics: rank-set size and baseline intersection probability. Small rank-set sizes are better - presumably as the confidence in ranks is better, and baseline intersection probability - large is better as it supposes the baseline method is closer to the true ranking. First: The paper should really report more standard ranking metrics such as precision/recall @ k, RBO, MAP or Normalized Discounted Cumulative Gain (NDCG). To factor in rank sets, you could use most probable ranking from your rank sets to compute these. Second: the absence of true rankings makes the empirical aspects less convincing. A synthetic experiment where true rankings are known perhaps would make sense to empirically demonstrate the main claims. - Use of machine labels just appear to widen the confidence bounds (e.g. Figure 3). The base conclusions on ranking appears to remain the same. In this instance, therefore, it remains unclear what the value of machine labeled information is. - Some additional insights into rank sets would add to the paper. For instance, how stable are rank sets to minor changes in $\mathcal{M}$? Uncertainty in rank for one model depends the overall set of models being considered, so this would be interesting to study. There are possibly other facets, such as how rank sets can be used in practice or the quality of the machine labeled data.

Questions

NA

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper tackles an interesting problem of evaluating ranking large language models automatically using a strong LLM as alternative to human preference estimates. The work primarily focuses on modelling uncertainty in such a ranking generated when compared to the distribution of human preference rankings. Since, pairwise comparison by humans are cumbersome and pairwise comparisons by strong LLMs are not completely consistent with human preferences, the authors propose a framework that improves upon pairwise ranking by strong LLMs. The authors propose a prediction power inference based framework to construct rank sets that provide coverage guarantees with respect to the true ranking consistent with human preferences.

Strengths

1. The work tackles an important problem concerning evaluation of ranking LLMs in an automated manner with respect to limited human preferences. The work discusses in details the drawbacks of existing ranking approaches and provides a statistically grounded framework (prediction powered inference) that works well in face of scarcity in human preference annotations. 2. The authors perform extensive evaluation on chatbot arena and propose two measures, namely rank set size and baseline intersection probability. 3. The proposed framework is useful for modelling uncertainty when using LLMs as judges and can also be applied to other scenarios such as modelling uncertainty in LLM driven relevance judgements for offline evaluation of retrieval.

Weaknesses

1. While the authors perform extensive evaluation, it might be a good idea to also test on other benchmarks like MT-bench or AlpacaEval related to approximation of human judgements. While authors already discuss the generalization aspect in limitations with regards to this, I would like to add it would also help address the concern regarding selection bias of test set and evaluators. Additionally, due to input limitations the benchmark may also not be representative of tasks that require reasoning over long form inputs and specifically complex reasoning tasks. Hence the leaderboard may only weakly correlate with real-world performance on these tasks. Additionally a minor point is that the metrics considered for rating response “relevance , helpfulness , accuracy , creativity and level” as shown in prompt might also change depending on the task: For instance when evaluating on a benchmark akin to QA tasks where precise information is needed creativity may not be a valid metric anymore. Hence evaluating on more benchmarks would help give a clearer picture on usefulness of the proposed framework. 2. While it is appreciated that the work provides the proof for theoretical coverage guarantees, the claim regarding the coverage guarantees made in Introduction and beginning of section 4 should be revisited as true rank-sets (true probabilities unknown) cannot be computed for LLMs. The Baseline intersection probability is a weak approximation for coverage guarantees. Though the authors argue that baseline method approximates well the true rank sets due to being constructed from large number of human pairwise comparisons this might not necessarily hold due to selection bias, distribution shift and various other factors. Without further evidence the claim that Baseline intersection probability is a good approximation for true coverage measure is not well supported.

Questions

Did the authors also try few-shot prompting the strong LLMs by showing few examples on how to judge the responses ? Would be interesting to see if this leads to any change in final observation and insights. For a small sample set would it be possible to empirically test coverage guarantees without the other metric being the proxy assumption?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Due to the inherent limitations of the benchmark, this work may also not be representative of tasks that require reasoning over long form inputs and specifically complex reasoning tasks.

Response to rebuttal

I appreciate the authors' responses to my questions and additional clarifications. Overall, I would like to keep the current score rating.