Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

Large Language Models (LLMs) have demonstrated promising capabilities as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. In this work, we first conduct a systematic study of the misalignment between LLM evaluators and human evaluation, revealing that existing calibration methods aimed at mitigating biases of LLMs are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and introduce Pairwise-preference Search (PAIRS), an uncertainty-guided search-based rank aggregation method that employs LLMs to conduct pairwise comparisons locally and efficiently ranks candidate texts globally. PAIRS achieves state-of-the-art performance on representative evaluation tasks in long-form generations and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the transitivity of LLMs and demonstrate how PAIRS benefits from calibration using debiased pairwise evaluations.

Paper

References (58)

Scroll for more · 38 remaining

Similar papers

Reviewer 6K4e7/10 · confidence 3/52024-05-09

Summary

This paper develops a method for automatically ranking a set of text responses (e.g., summaries of source texts) using LMs, while training to maintain alignment with human rankings. Problem background: Simply prompting an LM to score texts according to criteria can result in significant misalignment with human judgements. Estimating the marginal human score distribution p_H(s) and using that to calibrate the LM scores does not fully resolve the issue, implying that the model's likelihood p_M(y|s) is also misaligned from the human likelihood. Method: the proposed method (PAIRS) instead focuses on using pairwise comparisons between candidate texts to produce a global ranking. This is formulated as searching over an ordering (permutation) that maximizes the likelihood; the definition in Eq 4 is designed to considers two algorithms for finding this ordering. The first is a merge-sort-like algorithm that merges two sorted lists of responses by greedily choosing the next candidate from either list that locally maximizes the likelihood. They then introduce a beam-search-like alternative of this procedure that tracks the top-k beams of candidate rankings at any time. The experiments show that PAIRS achieves strong correlation with gold annotations on several natural language generation benchmarks.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

The proposed method is novel to my knowledge, seems relatively straightforward to implement, and the experiments appear to demonstrate strong empirical results.

Reasons to reject

* The text in 3.2 is not very clear at explaining how PairS-beam works, and the figure/algorithm need to be carefully studied to infer how it works without much supporting text. The caption or text could be expanded to make this easier to understand. * The results could benefit from more clarity about the actual real-world cost (in terms of # of model calls) of each method in the experiments. Do the proposed methods use more model calls than direct scoring?

Questions to authors

* What is the beam size used for PairS-beam(-scaled) in Figure 5? Does that matter at all for the number of model calls? * If I understand correctly, the scaling variant of Pairs is linear in number of samples to be ranked, is that asymptotically the same as direct scoring?

Reviewer VVGt9/10 · confidence 4/52024-05-11

Summary

This paper presents an LLM evaluation framework with pairwise comparisons (Qin et al. 2023). Pairwise comparisons require $O(N^2)$ evaluations where $N$ is the number of outputs from LLMs. So, this paper explores the use of transitivity of preferences to reduce the number of evaluations into $O(N \log N)$ and presents a sorting algorithm for LLM outputs, inspired by the merge sort algorithm. This study also proposes using beam search for merge operations because transitivity may not always hold. The experimental results demonstrate that the proposed method has higher correlations with human annotations and performs better than direct scoring and domain-optimized baselines.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

+ This paper is well-structured and written. + This paper proposes a reasonable algorithm for making pairwise approaches feasible for LLM comparisons. + The experiments demonstrate the superiority of the proposed approach, which has a great impact on LLM evaluation and development.

Reasons to reject

None

Questions to authors

The title includes "aligning with human judgment." I was expecting to see how the proposed method aligns LLM judgments with humans. However, this study does not directly align human judgment with that of an LLM evaluator. In fact, this paper gave up calibrating LLM evaluators directly for alignment, but obtained a better alignment as a result of utilizing pairwise comparisons. So I felt that this phrase in the title is a bit misleading. Is it right that evaluator LLMs were not trained to align human pairwise preferences? Where can I find the numbers of pairwise comparisons of the PAIRS-greedy, PAIRS-beam, and Qin et al. (2023)?

Reviewer zw924/10 · confidence 4/52024-05-13

Summary

This paper develops a search-based search algorithm called PAIRS, which searches example pairs for LLM pairwise judgment to create a global ranking of examples. The basic idea is to convert the pairwise judgments conducted so far to prune the pairs for which the judgements can be inferred. Experiment results show that PAIRS can produce judgments that are more correlated with human judgments than a scoring-based method G-Eval.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

- (S1) Aggregating pairwise LLM judgments into a global ranking is an interesting idea and not well studied. LLM-as-a-Judge is an active research area and the paper could contribute to the research community. - (S2) The paper is generally well-written.

Reasons to reject

- (W1) Baseline methods are missing. It is not clear if the proposed solution is reasonably effective for the purpose. More specifically, on (1) search and (2) aggregation. For (1), the most naive way is to use randomly selected pairs with the same rank aggregation method. For (2), other aggregation methods such as the Elo score, which is commonly used for LLM ranking (eg., Chatbor Arena). - (W2) The motivation for Evaluation as Ranking is unclear. The main purpose of evaluation in most cases is to compare two (or multiple systems) to judge which one is better (best). In that sense, the global ranking discussed in the paper may not be the best option for evaluation.For example, the SummEval dataset used to 1-5 ratings for manual evaluation and the granularity is much more fine-grained by the proposed method. - (W3) The cost of the proposed method is expensive. The computational complexity for scoring-based methods such as G-Eval is O(N). It is not clear if it’s worth paying extra cost. One option is to conduct experiments under the same budget (ie., just conducting N pairs to be fair against scoring-based methods) and discuss if the pairwise judgment approach can be more effective. - (W4) Section 2 is misleading, as the limitations discussed in the section are not addressed by aggregating pairwise judgments. If it is, the paper needs to show that the rating distribution is better aligned with that of human judgments (which is not straightforward, as the judgment is global ranking)

Questions to authors

Please respond to the concerns raised in Reasons to Reject.

Authorsrebuttal2024-06-07

Rebuttal 2

We thank the reviewer for the feedback and have addressed them below. 1. Pairwise Baselines This ablation study with other two pairwise baselines using a different format than our result Table 1. In this table, 50, 100, 240(max) represents the number of randomly selected candidate pairs used to calculate the win rate or the ELO rankings. For SummEval dataset, there are 16 candidates for each datapoint, resulting in a total of 240 pairs. (N^2, including swapped comparisons and excluding self-comparison.) For PairS-Greedy, we cannot control the exact number of comparisons needed for the ranking. By averaging the results over 10 repetitions, we use 39.2 and 44.3 comparisons to achieve the global rankings for Mistral-7B and GPT-3.5. When using calibrated PairS-Greedy, (where each comparison uses the average probabilities of pairwise comparison with permutated positions), the number of comparisons are 41.5*2 and 46.3*2. The reason we mention calibrated result is that the 240 total comparison pairs include comparisons with swapped positions. Both uncalibrated and calibrated PairS-Greedy, our method are significantly more efficient than the win rate and ELO baselines. Presenting these results in a table may not be optimal, and we plan to use line figures in the formal revision to better illustrate the findings. --- 2. Ranking as evaluation is not motivated > If you compared M systems for N pairs, wouldn’t the computational complexity be O(M^2 * N^2)? We believe the reviewer has misunderstood the setup of pairwise ranking. **We are not comparing the whole LLM systems, but a list of candidates.** The candidates can be generated by different systems, but we only evaluate the quality of those candidates. If we have M candidates to rank, we can sample up to M*2-M pairs. What we meant by "as the number of candidates is typically much larger than the number of score categories", is M is usually larger than the scale of score categories (5 for likert scale). Therefore, providing the ranking of M candidates is more fine-grain than scoring. --- 3. pointwise evaluation and G-Eval We acknowledge the statement in G-Eval paper. We believe the reason why G-Eval cannot handle API model efficiently is beyond the discussion of our paper. In our paper, what we claim is pointwise evaluation methods generally have complexity of O(N) and our argument is still hold: pointwise evaluations can only gain marginal performance with more computations. --- In conclusion, we believe our experiments and ablation studies provide a comprehensive evaluation of the proposed PairS method. We have compared with multiple baselines including both pointwise and pairwise evaluation methods. Our results show that PairS is more efficient and effective than pairwise baselines and achieves higher human agreement than both pointwise and pairwise baselines. Furthermore, we have clarified and provided explicit support for the motivation behind ranking evaluation. We also have address reviewer's misunderstanding regarding our discussion of the limitation of direct scoring in Section 2. We have made a sincere effort to respond to the reviewer's concerns and to provide additional evidence and clarifications to support our claims. In light of our response, we respectfully request that the reviewer consider improving their score, as we believe our work makes a valuable contribution to the field of LLM evaluator.

Reviewer VjHC7/10 · confidence 4/52024-05-24

Summary

- This paper proposes a pairwise-preference search method that estimates MLE of preference rankings by searching through an uncertainty-pruned pairwise comparison space. - The proposed method is more effective than traditional direct scoring evaluations and existing pairwise baselines.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The writing is clear to understand. - The analysis of the limitations of calibration in aligning LLM evaluators is well presented and the proposed PAIRS method is also well described. - Experiments are achieved with open-source and close-source models on multiple datasets. - Ablation studies are conducted.

Reasons to reject

- The main LLM evaluation results with the proposed method are not always better than the direct scoring method, which makes me doubt about the motivation in 3.1. - Some results in Table 1 are close to the baselines. Are those statistically significant? - Experiments can be more comprehensive if different size of open-source models (e.g. 70B size models). Also, it is curious if the larger models perform similarly.

Questions to authors

- Any reason for the beam size of 1000, which seems to be very large number? - What if the beam size is smaller for the main experiments? Also, what about the correlation between the inference time and the different beam sizes?

Authorsrebuttal2024-06-04

Following up discussion towards the rebuttal

We thank the reviewer again for the valuable suggestions. We hope our complementary experiments have addressed all the reviewer’s concerns. We hope that in light of the response, the reviewer could consider improving their score. We also provide some further discussion to support our rebuttal --- > Pairwise Baselines PairS leverages the transitivity assumption of LLMs, which gives it a natural efficiency advantage. Win-rate and ELO aggregation methods are designed to handle systems or players with non-constant performance, making them more effective at dealing with changes. However, in our case, we are ranking the exact same aspect of constant response candidates, which means sorting-based algorithms that exploit transitivity have inherent benefits. --- > Ranking as evaluation is not motivated To support our motivation claim in Sec 3.1, we conducted a complementary experiment using 50 randomly sampled summary candidate pairs for each source text in SummEval. We calculated the 3-way agreement rate accuracy between direct scoring and human scores, as well as between pairwise comparisons and human scores. The results consistently show that pairwise comparisons have better agreement with human judgements than scoring, across all three selected LLMs. |Model|Scoring|Pairwise| |-|-|-| |Mistral-7B|38.7|45.5| |Llama2-7B|39.6|43.0| |GPT-3.5| 42.3|48.1| Moreover, ranking-based evaluation has already been successfully employed in various real-world applications, such as re-rankers for data selection, output reordering, and recommendation systems. --- > PairS may be expensive More discussion about the cost of pointwise methods, such as G-Eval. G-Eval uses a weighted average of 20 inference repetitions, resulting in total computations of 20N. In comparison, PairS-greedy requires Nlog2(N) in the worst case. Specifically, for the 16-candidate SummEval example mentioned above, the factor is only 2.5N.

Reviewer zw922024-06-07

Thank you for your responses and for sharing additional experiments. Apologies for the delayed reply. Here are my follow-up questions for each of the points in your both responses. >> Pairwise Baselines > > Uncalibrated PairS-greedy achieves the same performance as the baselines using only 34% and 31% of the comparisons. Calibrated PairS-greedy, which doubles the number of comparisons, can sometimes outperform baselines that use O(N^2) comparisons. Thank you for sharing the additional experiment results and the observations. I cannot draw the conclusion from the table. Could you add an explanation for the table in a similar style as the paper? What are these numbers? What are 50, 100, 240(max)? What do 23.0(39.2) and 36.5(41.5*2) mean for PairS-G? >> Ranking as evaluation is not motivated > > as the number of candidates is typically much larger than the number of score categories If you compared M systems for N pairs, wouldn’t the computational complexity be O(M^2 * N^2)? Can N pairs be sampled from M systems and can offer the same level of ranking quality as that for 2 systems? I think this point is not discussed or clarified in the paper. >> PairS may be expensive > > We acknowledge that pairwise evaluations are generally more expensive than pointwise methods. However, we would like to highlight: > > 1. Performance should be considered alongside cost. We show that PairS outperforms most pointwise methods, such as G-Eval, which only obtain marginal gains with increased computation. > 2. PairS is highly efficient compared to pairwise baselines. > G-Eval uses a weighted average of 20 inference repetitions Thank you for sharing the additional experiment. The G-Eval paper called the API multiple times because the GPT API does not return token probababilities. The method uses the weighted average of tokens (e.g., 1-5) based on their probabilities, which can be computed by a single inference. Below is the description from the G-Eval paper. ``` For GPT-4, as it does not support the output of token probabilities, we set ‘n = 20, temperature = 1, top p = 1’ to sample 20 times to estimate the token probabilities. ``` >> The limitation in Section 2 > > It is important to note that this paper does not focus on calibration, Thanks for clarifying. Then, I believe Section 2 is misleading and the point should be further clarified. This is rather a presentation issue and I don't have any follow-up questions.

Authorsrebuttal2024-06-04

Following up discussion towards the rebuttal

We thank the reviewer again for the insightful feedback. We also provide some further discussion on the performance on Larger LLMs: > Performance on Larger LLMs While computational resources prevent us from conducting larger-scale evaluations, we additionally include results from Llama2-13B, and we will endeavor to include larger-scale LLMs in future work in addition to the large-scale GPT4 experiments that we have conducted : ||PairS-greedy| |-|-| |Llama2-7B|15.3+-1.6| |Llama2-13B|17.9+-1.1|

Reviewer VVGt2024-06-05

Re: Rebuttal by Authors

Thank you for the response. It's my pleasure to contribute to the work in some way through the reviewing process.

Authorsrebuttal2024-06-05

Thanks

Thank you for your kind message. We truly appreciate the time and effort you have put into reviewing our work. Thank you for your constructive feedback and for being a part of this collaborative effort to advance our field of study. Best regards, Authors

Authorsrebuttal2024-06-05

Reminder

Dear Reviewer VjHC, We thank you once again for your feedback! As the discussion period is ending, we wonder if you could take a look at our response if you haven't? We are eager to know whether our response has addressed your concerns: we'd be grateful if you could improve the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have. Best, Authors

Reviewer VjHC2024-06-05

Thanks

Thank you to the authors for their response, the additional ablation on beam size and the result of different size are helpful. I am still curious of inference speed vs number of beam sizes but my questions have mostly been clarified.

Authorsrebuttal2024-06-06

Thanks

We thank the review for confirming all questions have been answered. As for the inference speed, the numbers in the bracket are averaged numbers of model forward calls. We believe it can represent the inference speed. The actual inference time depends on the LLM and GPU used. For example, in our case, it takes about 7s to perform a PairS-greedy (around 40 model calls) with Mistral-7B and batch size of 5, using RTX3090.

Authorsrebuttal2024-06-05

Reminder

Dear Reviewer zw92, We thank you once again for your feedback! As the discussion period is ending, we wonder if you could take a look at our response if you haven't? We are eager to know whether our response has addressed your concerns: we'd be grateful if you could reconsider the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have. Best, Authors

Authorsrebuttal2024-06-05

Reminder

Dear Reviewer 6K4e, We thank you once again for your feedback! As the discussion period is ending, we wonder if you could take a look at our response if you haven't? We are eager to know whether our response has addressed your concerns: we'd be grateful if you could improve the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have. Best, Authors

Reviewer 6K4e2024-06-05

Thank you to the authors for their response, the additional ablation on beam size is helpful. I feel that my questions have largely been clarified and have no further questions.

Authorsrebuttal2024-06-06

Reminder

Dear Reviewer zw92, As the discussion period is ending, we would like to remind you to take a look at our response if you haven't. We are eager to know whether our response has addressed your concerns: we'd be grateful if you could reconsider the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have. Best, Authors

Program Chairsdecision2024-07-10

Decision

Accept

© 2026 NYSGPT2525 LLC