PiCO: Peer Review in LLMs based on the Consistency Optimization

Summary

The paper introduces a novel unsupervised evaluation method for large language models (LLMs): it uses a peer-review mechanism of a models' anonymized answers by other models. The approach assigns a (learnable) capability parameter to each LLM and solves a constrained optimization problem to maximize the consistency between capabilities and scores. The end result is a ranking of the evaluated models.

Strengths

The paper introduce a novel approach to an important practical problem: ranking the quality of the ever-growing number of open- and closed- source LLM available to the public. The paper is reasonably easy to follow, and the empirical results appear to be sound.

Weaknesses

The paper could be further improved on several directions: 1) you should dedicate a full section to the iterative elimination of models; what is the benefit of eliminating the weaker ones rather than keeping them around? how di you come up with the threshold of 60% to remove? can you learn this threshold automatically? is this threshold optimal for these 15 models? what happens if you start with, say, 100 models? what happens to your results (and the curves in Fig 5) if you stop earlier (all three metrics, not just CIN)? What if you continue to eliminate all models until you are left with one? is there any relationship between the order in which the models are eliminated and their final rank? 2) are there any scaling issues for 100, 1K, 10K, or 100K models? how about cost: is it cheaper to fine-tune a "baseline" model than to pick the best one out of 10K candidates? 3) while the three metrics you use are meaningful, you should also present results for Precision@1 and -say- RBP@3; after all, we care a lot about identifying the the top models 3) Fig 5 should be extended to nine graphs (3 metrics * 3 datasets); for each of the 9 graphs, you should also show illustrative three ranked lists: PiCO's, PRE's, and the target one. As always, the devil is in the details: not all "CIN = 1" are created equal. Performance-wise, it is almost irrelevant if you have the bottom-2 models inverted; no necessarily so if the inversion is between the top-2 models

Questions

1. line 144 - what is the benefit of a model evaluating itself? why not explicitly disallow it? w/o a full explanation, this detail unnecessarily complicates the story 2. you repeatedly talk about "higher-level LLM" w/o defining the concept; is it based on the number of parameters (larger --> higher)? 3. can PiCO (or an extension) also estimate the gap between the performance of two models that are adjacent in the final ranking? NITPICKS - line 72: "closer" ... than whom? existing approaches, I presume - line 154: the "quadruple" appears to be a "triple"; adding "s" would make it a 4-tuple - you talk a few times about "persuasiveness;" this concept makes sense for human reviewers discussing a submitted article, but it is unclear why it matters for LLMs - the related work section should be right after the Intro

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper studies how to estimate LLMs' performance ranking without human preference annotations. In particular, it proposes to leverage three metrics (PEN, CIN, LIS) to evaluate the estimation quality, gives an estimation mechanism that first asks a list of LLMs (called "reviewers") to rank pairwise answers to user questions independently, and then aggregates their ranking via a weighted sum approach. A consistency optimization determines the weights of each reviewer.

Strengths

The problem of LLM evaluation without human annotations is critical in resource-limited applications. The most important and interesting contribution of this paper, in my opinion, is proposing the problem of estimating the performance rank of LLMs instead of any metric of an individual LLM. The paper also reveals an interesting assumption that better reviewers are expected to be better answer generators, which leads to their consistency optimization approach. Overall, the paper is well-written and easy to follow.

Weaknesses

While I find the proposed problem interesting, there are still a few limitations, unfortunately. ***Unclear implication of ground truth ranking***: The technical part of the paper starts by introducing a ground truth ranking (equation (1)) without giving its physical meaning. It simply assumes "[...] alignment with human preferences", but it is not clear what human preferences mean in this context. ***Evaluation metric is strange***: One of my major concerns is on the choices of evaluation metric. All the three proposed metrics, PIN, CIN, LIS, in the authors' own words, seem originally used for time series comparison. However, the goal here is to compare rankings, not time series. Thus, it is unclear why we should not use the standard ranking comparison metrics, e.g., Spearman's rank correlation coefficient or Kendall rank correlation coefficient. ***Consistency optimization algorithm is not provided***: The core of the proposed ranking estimation method is the optimization problem (7). It does not seem to be a standard optimization problem, but I could not find (even a discussion on) any clue on how to solve it in this paper. ***An optimal solution to the consistency optimization formulation can be useless***: I find the following optimal solution to the problem (7): just set weight w to be 0 for all LLMs. It is an optimal solution as G and w are identical and thus the objective is always maximized. However, this solution is undesired. I probably misunderstood something, but this seems to suggest the formulation is incorrect. ***Consistency optimization formulation seems brittle to query distribution biases***: Another problem with the formulation is that it seems brittle to data distribution bias. E.g., suppose M1 is indeed better than M2 for some query q. And let us replicate many copies of q in the dataset D. Then the grade G1 can be arbitrarily large. In other words, the grade of an LLM is proportional to the number of battles involving it in the dataset D, which should not be the case. ***Choices of LLMs for evaluation***: In line 547, the authors write "For our analysis, we meticulously selected 15 LLMs". What is the principle of the meticulous selection? Other than open-source and close-source, the selection is quite arbitrary. For example, I am quite surprised to not see GPT-4 and Claude included in the reviewer LLMs. ***Comparison with a simple baseline***: One simple baseline is to ask a powerful LLM(e.g., GPT-4, Cluade-3) to give a preference for each answer question pair, and then take the vote to determine the ranking. I would suggest to compare the proposed method with this simple baseline.

Questions

***Unclear implication of ground truth ranking***: What is the physical meaning of the ground truth ranking? What does that imply? ***Evaluation metric is strange***: What is the motivation of using time series comparison metrics to compare rankings? How does the proposed method work under ranking comparison metrics? ***Consistency optimization algorithm is not provided***: How do you solve problem (7)? ***An optimal solution to the consistency optimization formulation can be useless***: How do you prevent the useless but optimal solution? Does that imply the formulation is incorrect? ***Query distribution and LLM judgment biases***: How could this formulation resist query distribution and LLM biases? ***Choices of LLMs for evaluation***: What is the principle of the meticulous selection? How does the proposed approach work if strong models such as GPT-4 and Claude are included in the reviewer LLMs? ***Comparison with a simple baseline***: How does the proposed method compare with the simple baseline?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

No. The limitations are not well discussed.

Summary

In this paper, the authors propose a more reliable evaluation system to rank the abilities of different large language models (LLMs). Previous evaluation methods typically suffer from two main drawbacks: (1) benchmark data leakage and (2) cost-intensive and potentially biased human evaluations. To address these issues, the authors introduce an unsupervised evaluation mechanism to measure the abilities of LLMs and derive their rankings. The core idea of this mechanism is to first collect the answers from each LLM, then treat each LLM as a 'reviewer' to rate the quality of the other LLMs' answers, and finally optimize the internal agreement of the reviews among all LLMs. They also conduct experiments on three datasets to validate the effectiveness of their proposed mechanism.

Strengths

1. Unsupervised Evaluation Method: The paper introduces PiCO (Peer Review in LLMs based on Consistency Optimization), a new unsupervised evaluation method that leverages peer-review mechanisms to measure the capabilities of LLMs automatically, particularly without human-annotated data. The unsupervised nature also makes it scalable and less subjectively biased. 2. Consistency Optimization Framework: The proposed approach includes a constrained optimization method based on the consistency assumption, which helps in re-ranking LLMs to align more closely with human preferences. 3. New Evaluation Metrics: The paper proposes three new metrics—Permutation Entropy (PEN), Count Inversions (CIN), and Longest Increasing Subsequence (LIS)—to evaluate the alignment of LLM rankings with human preferences. These metrics can further inspire future work.

Weaknesses

1. Reliance on Consistency Assumption: The effectiveness of the method relies on the consistency assumption that higher-level LLMs can more accurately evaluate others. However, a natural concern is, "Does this assumption always hold true in practice?" I suggest the authors further discuss the applicability of their method. 2. Complexity of Implementation: The framework involves a complex review process, which requires substantial computational resources to support the LLMs' inference. Can you provide some details on the number of tokens consumed and a comparison of consumption with baseline methods? 3. Consideration of Multi-Agent Methods: Since the proposed method employs multiple LLMs, I think more recent and advanced evaluation methods based on multi-agent systems should also be considered in the experiments, such as AgentVerse. 4. Details of ELO: The ELO system is essential for the proposed mechanism. However, it is only mentioned in the appendix. I suggest the authors add more details about the background of ELO and how it is adopted in the proposed mechanism.

Questions

Please see my above comments

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

1

Limitations

Yes

Summary

The highlight of the work is the proposed method of evaluating Large Language Models without relying on human feedback.

Strengths

- The proposed evaluation method is a novel attempt of automating the LLM improvement process. Such method worth further exploration. It could be adapted to many of the LLMs and potentially bring us more insights. - By eliminating the involvement of human, the proposed evaluation method limits the bias brought by human labelers. The observations presented are also interesting as LLMs can sometimes surprise us. - Great presentation and visualization.

Weaknesses

Some of the equations and notations in the paper seems unnecessarily complicated, which can be reorganized when polishing.

Questions

1. How do you define unlabeled questions in line76, as in to what extend is a question unlabeled? 2. How do you differentiate high-level from low-level LLMs? Are they interchangeable or more of a dynamic definition while evaluating on different tasks?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The idea is straightforward and make sense to me. But LLMs can be trained to bypass such systems, which may lead to potential fairness or security problems.

Many thanks to the authors for the detailed answers. After considering both the other reviews and the rebuttals, I maintain the original rating.

Many thanks to the authors for the detailed answers. After considering both the other reviews and the rebuttals, I maintain the original rating.