Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement

We thank the reviewer for the positive comments! We are glad that you view our work to enhance the reliability of LLM-based evaluations.   ### Is this a theoretical work? While our suggested framework is grounded on the theories of inferential statistics, we believe our core contribution lies on its significant empirical results, that make LLM judges to be more reliable and (simultaneously) more cost-effective. We are happy to see that the reviewer also recognizes the framework’s effectiveness in improving LLM judges across different tasks.   ### User-friendly toolkit We appreciate this suggestion! We are already working on implementing a high-level interface of our method where users only have to put in the name of judges and target agreement level. The framework then would automatically (1) calibrate the evaluation confidence, and (2) decide the abstention policy to provide the human agreement guarantee. We hope this addresses your comments!

Dear reviewer mWkk, We have updated the submission and addressed the feedback from each reviewer. We are checking in to see if we have adequately addressed your concerns and whether you would consider adjusting the paper's rating. Thank you,

Thank you for the positive endorsement of our work. We are happy that you found our work to be well-written and structured, our experiments to be meaningful, and overall to clearly improve LLM-based evaluation!   ### On future extension of Cascaded Selective Evaluation As noted in the conclusion, we focused on pairwise evaluation as it represents the most widely used applications of LLM judges, but in principle, our framework can be generalized beyond the pairwise format. We are excited to work on the future extension of Cascaded Selective Evaluation for diverse evaluation scenarios, such as Likert Scale.   ### Qualitative analysis on abstained examples We appreciate this suggestion. We followed your comments to analyze the abstained examples in a more qualitative perspective, by (1) categorizing prompts based on their contents and identifying characteristics of abstention policy, (2) evaluating whether humans would have “abstained” for the same samples with the LLM judges. We’ve included 8 qualitative examples across summarization evaluation / chat assistant evaluation **(Appendix C)**, which would help making it clearer the insights from the qualitative analysees. We will make a clearer pointer to these examples in the main section. ***Content-based categorization*** \ We first categorized prompts based on their contents and checked if there’s distinctive characteristics of model abstention policy. We followed [1] to divide prompts in ChatArena into the following 8 types - Advice, Communication (ChitChat), Writing, Reasoning, Knowledge-Aware, Unsafe Query, NLP Tasks, and Others, and used GPT-4o to automatically categorize each prompt into one of the 8 types. Then, we measured (1) the average human inter-annotator agreement, (2) ratio of machine-abstained samples, (3) the ratio of samples evaluated by each judge model in the cascades per each category. | | Advice | Communication (chitchat) | Writing | Reasoning | Knowledge Aware | Unsafe Query | NLP Tasks | Others | |:-----------------------------:|:------:|:------------------------:|:-------:|:---------:|:---------------:|:------------:|:---------:|:------:| | Human IAA | 0.78 | 0.76 | 0.78 | 0.85 | 0.76 | 0.92 | 0.78 | 0.81 | | Machine Abstention Rate | 0.52 | 0.51 | 0.53 | 0.50 | 0.55 | 0.24 | 0.60 | 0.46 | | Mistral-7B Evaluation Rate | 0.18 | 0.19 | 0.08 | 0.15 | 0.10 | 0.56 | 0.15 | 0.20 | | GPT-3.5-turbo Evaluation Rate | 0.24 | 0.22 | 0.23 | 0.15 | 0.20 | 0.16 | 0.15 | 0.24 | | GPT-4-turbo Evaluation Rate | 0.06 | 0.08 | 0.16 | 0.20 | 0.15 | 0.04 | 0.10 | 0.10 | The results are as shown above. While there is no salient trend across the categories, one notable outlier in machine abstention rate is in Unsafe Query, with significantly lower abstention rate (0.24) than others. We posit that the instruction-tuning process that incorporates safety training could have led the LLM judges to be confident in their evaluation for choosing a safer choice for unsafe query. An example of this category can be found in Chat Assistant Example 3 in the appendix. Meanwhile, we find that GPT-4-turbo tends to be more involved in reasoning type of prompts than e.g. ChitChat, which follows our intuition that harder tasks should require stronger judges in general (an example is provided in **Chat Assistant Example 2 in Appendix C**). Overall, we generally find that the machine-abstained examples tend to require subjective evaluation in nature, e.g. asking for a preference between two jokes **(Chat Assistant Example 6 in Appendix C)**. [1] Julong Li et al. Dissecting Human and LLM Preferences. 2024.

We thank the reviewer for both positive and helpful comments! We are excited to see that you found our paper to tackle an important issue, well-written, and our experiments and ablations to be insightful! Below are responses to each of your suggestions:   ### Stronger connection to Selective Prediction Thank you for the point.The objective of cascaded selective evaluation is indeed comparable to that of selection with guaranteed risk (SGR) [1] in the selective prediction literature, while the specific method differs in how to operationalize this guarantee. We formulate the selection of abstention thresholds as a multiple-testing problem, compared to Bonferroni Correction in SGR, allowing better coverage while providing the same guarantee **(Section 2.1)**. We also generalize the selective prediction guarantee to cascades of models, rather than a single judge model **(Section 2.3)**. We will clarify the link between selective evaluation and selective prediction, along with their guarantees. [1] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks, 2017.   ### Cheaper Judges are still costly? While the judge models with 7B parameters may not be the smallest possible model, we believe this is still a notable result, given the scale of judge models widely used in the literature. For example, in a recent paper that employs a suite of “small” juries instead of a single judge model [1], the smallest LLM employed as a jury is 35B (aside to other models used in parallel). Another paper [2] employs a 7B judge model as in our experiments, but the model has been fine-tuned with specialized synthetic dataset to improve its evaluation capability, while we employ an off-the-shelf model to maximize the framework’s utility and generalization. Even the strongest judge model GPT-4 fails to consistently align with humans, e.g. marking only 63.2% agreement in Auto-J **(Figure 5). On the contrary, our work aims to improve the alignment of LLM judges even without fully relying on the frontier models, thus making automatic evaluation more cost-effective and scalable.** That said, I’m excited to see even smaller judges than 7B-scale that stay competitive with their larger counterparts, which could make larger presence in the cascaded framework. [1] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating llm generations with a panel of diverse models, 2024. [2] Seungone Kim et al. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models, In Proceedings of EMNLP 2024.   ### Removal of disagreement signals While the evaluation of our framework is based on how well it predicts the majority preference label, this only serves for evaluation purposes (mainly due to the meta evaluation benchmarks evaluating majority alignment - e.g. both ChatArena and Auto-J provide majority human preference label only). **In fact, since our framework is objective-agnostic, it can simply be extended for individual alignment by changing what it’s supposed to guarantee (from agreement ratio with majority label to agreement ratio with individual preferences).** Specifically, one way to operationalize this is to collect a calibration set from a consistent set of human evaluators, and align the model judgement to match a specific annotator’s preference. In this regard, we believe Cascaded Selective Evaluation can serve as a useful tool for modeling personalized preferences with otherwise monolithic LLM judges, rather than removing useful individual preference information.   ### What happens to Human IAA when we lower target agreement level? When we set target human agreement $1-\alpha$ to 0.8 instead of 0.9, the average human IAA for abstained samples goes from 0.815 to 0.806, i.e. only the harder-to-reconcile samples tend to be abstained by the models. This is due to the fact that when we lower the target agreement level, the evaluation coverage improves, hence the model tends to abstain less (only abstain when they sufficiently lack confidence).   ### Related Works We appreciate the suggestions, these works are indeed relevant to improving reliability of LLM-based evaluation. We will include these works in our revision.   We hope the responses above address your questions - thank you again for the constructive and positive feedback!

We sincerely thank the reviewer for the positive feedback and great suggestions ! We are glad that you found our work to have clear methodological contributions and our experiments to be very carefully executed. Below, we address each of your comments:   ### Cost comparison between confidence estimation methods Thank you for the comment. Following your suggestion, we compared the resource requirements of Simulated Annotators and canonical methods on AlpacaEval. We focus on the cost requirement (as measured by relative API cost), since the majority of considered methods can be parallelized to generate multiple generations with LLM – e.g. Semantic Set, Semantic Entropy – and thus their wall clock runtime wouldn’t be significantly different from Predictive Probability. | | ECE ↓ | AUROC ↑ | Relative API Cost | |------------------------------|:---------:|:---------:|:-----------------:| | Predictive Probability | 0.374 | 0.457 | 1.000 | | Predictive Probability (CoT) | 0.292 | 0.527 | 20.376 | | Verbalized Confidence | 0.414 | 0.490 | 2.215 | | Semantic Set | - | 0.513 | 20.376 | | Semantic Entropy | - | 0.572 | 20.376 | | Simulated Annotators (N=K=3) | 0.089 | 0.613 | 3.513 | | Simulated Annotators (N=K=5) | **0.075** | **0.632** | 6.910 | The results are as shown above. Simulated Annotators under both settings outperform all canonical methods in terms of ECE and AUROC, while incurring significantly less cost than more sophisticated methods (Predictive Probability (COT), Semantic Set, Semantic Entropy). We’d like to also emphasize that while calibrating Simulated Annotators does make a tradeoff between calibration and cost compared to Predictive Probability (or zero-shot evaluating with LLM Judge with considering their confidence), the higher cost can often be offset under the cascaded framework by dynamically selecting cheaper judge to evaluate the sample.   ### Controlling coverage for baselines To remove the effect of confounders, for all baselines in Table 2 and 3, we used the same cascades of models and the same confidence measure (Simulated Annotators) as Cascaded Selective Evaluation — the only difference between our method and the considered baselines is in how they decide the abstention policy $\hat{\lambda}$ (and therefore the coverage of each LLM judge). Therefore, if we fix the coverage in a baseline with the value we obtained via Cascaded Selective Evaluation (e.g. in Table 2, enforcing Mistral-7B to evaluate 28.3% * 55.7% of the samples, GPT-3.5 to evaluate 28.2% * 55.7% of the samples, …), the baseline would be essentially equivalent to Cascaded Selective Evaluation. **The results on Table 2 and 3 show that even with the same judge models and confidence measure, the reliability of evaluation can be drastically different depending on how we set the abstention policy.** Unlike the baselines, Cascaded Selective Evaluation precisely controls $\lambda$ that guarantees target human agreement in the unseen set of samples. That said, we are excited to see future improvements on Cascaded Selective Evaluation that adopt even better confidence measures and better cascades of judge models. As you noted, given that the machine abstentions correlate with the subjectivity perceived by humans, perhaps a future work could explore modeling subjectivity with a multi-agentic system to better elicit hard-to-reconcile samples.   ### Going further with abstained samples Thank you for the great suggestion! Below, we followed your suggestion to further augment the weaker cascades by adding GPT-4-turbo as the last judge. | | Empirical Human Agreement | Coverage | Guarantee Success Rate | Relative API Cost | |:-------------------------:|:-------------------------:|:--------:|:----------------------:|:-----------------:| | *weaker cascades* | 80.3 | 68.3 | 90.8 | 0.126 | | *weaker cascades + GPT-4* | 80.4 | 78.2 | 90.6 | 0.192 | Notably, the augmented cascades lead to substantially better coverage while providing the same guarantee as before. At the same time, the cost is still much less than relying solely on GPT-4 (1.000).

### Does the composition of judges correlate with the difficulty of the task? Yes, intuitively if the evaluation task is hard (e.g. evaluating generations for hard reasoning prompts), then the task would generally require stronger judges to better align with the ground truth. While this is generally true, the real challenge lies in that it is hard to predict (in advance) how strong of a judge we need to use to reliably evaluate a given task. In this sense, the core contribution of our framework is that it provides a way to select when to trust which judge model, while not worrying about the precision of the evaluation. In an ideal scenario, the evaluation wouldn’t incur any cost for API calls, as Mistral-7B would evaluate all samples without abstention. The worst case scenario for the cost of model cascades would be equivalent to using GPT-4 for full evaluation, so long as the baseline uses the same prompting strategy. However these scenarios are extreme, and most results will interpolate between the two, as shown in the results in **Table 2, 3, and 8**. We will include the discussion on the relationship between task difficulty and judge composition in Section 3.6.   ### Consistent $1-\alpha$ Thank you for notifying this, the target accuracy in the tables were chosen based on the difficulty of the evaluation task, as measured by GPT-4’s average human agreement level in each dataset. We think this way best compares the results against relying on GPT-4 in each benchmark, the most widely adopted evaluation scheme in the literature. However, we also detailed results across different levels of $1-\alpha$ in each accompanied figure **(right side of Figure 3, 4 and 5)**. | | Empirical Human Agreement | Coverage | Guarantee Success Rate | Relative API Cost | |:-----------------:|:-------------------------:|:--------:|:----------------------:|:-----------------:| | *weaker cascades* | 85.7 | 59.5 | 90.5 | 0.163 | Here, we additionally report the results for weaker cascades in Table 7 but with $1-\alpha = 0.85$, consistently with the setup of main experiments on ChatArena in Table 3.   Again, thank you for your positive and constructive suggestion! We will clarify the rationale behind our choice of \alpha in a footnote.