Rebuttal 2
We thank the reviewer for the feedback and have addressed them below.
1. Pairwise Baselines
This ablation study with other two pairwise baselines using a different format than our result Table 1. In this table, 50, 100, 240(max) represents the number of randomly selected candidate pairs used to calculate the win rate or the ELO rankings. For SummEval dataset, there are 16 candidates for each datapoint, resulting in a total of 240 pairs. (N^2, including swapped comparisons and excluding self-comparison.)
For PairS-Greedy, we cannot control the exact number of comparisons needed for the ranking. By averaging the results over 10 repetitions, we use 39.2 and 44.3 comparisons to achieve the global rankings for Mistral-7B and GPT-3.5.
When using calibrated PairS-Greedy, (where each comparison uses the average probabilities of pairwise comparison with permutated positions), the number of comparisons are 41.5*2 and 46.3*2. The reason we mention calibrated result is that the 240 total comparison pairs include comparisons with swapped positions.
Both uncalibrated and calibrated PairS-Greedy, our method are significantly more efficient than the win rate and ELO baselines. Presenting these results in a table may not be optimal, and we plan to use line figures in the formal revision to better illustrate the findings.
---
2. Ranking as evaluation is not motivated
> If you compared M systems for N pairs, wouldn’t the computational complexity be O(M^2 * N^2)?
We believe the reviewer has misunderstood the setup of pairwise ranking. **We are not comparing the whole LLM systems, but a list of candidates.** The candidates can be generated by different systems, but we only evaluate the quality of those candidates. If we have M candidates to rank, we can sample up to M*2-M pairs. What we meant by "as the number of candidates is typically much larger than the number of score categories", is M is usually larger than the scale of score categories (5 for likert scale). Therefore, providing the ranking of M candidates is more fine-grain than scoring.
---
3. pointwise evaluation and G-Eval
We acknowledge the statement in G-Eval paper. We believe the reason why G-Eval cannot handle API model efficiently is beyond the discussion of our paper. In our paper, what we claim is pointwise evaluation methods generally have complexity of O(N) and our argument is still hold: pointwise evaluations can only gain marginal performance with more computations.
---
In conclusion, we believe our experiments and ablation studies provide a comprehensive evaluation of the proposed PairS method. We have compared with multiple baselines including both pointwise and pairwise evaluation methods. Our results show that PairS is more efficient and effective than pairwise baselines and achieves higher human agreement than both pointwise and pairwise baselines.
Furthermore, we have clarified and provided explicit support for the motivation behind ranking evaluation. We also have address reviewer's misunderstanding regarding our discussion of the limitation of direct scoring in Section 2.
We have made a sincere effort to respond to the reviewer's concerns and to provide additional evidence and clarifications to support our claims. In light of our response, we respectfully request that the reviewer consider improving their score, as we believe our work makes a valuable contribution to the field of LLM evaluator.