Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

We would like to thank the reviewer for their encouraging evaluation of the paper. Below, we respond to the reviewer’s queries raised on paper weaknesses/questions. For context, we first talk about key takeaways. **1. Some key takeaways -** * Advise against uncritical reliance on Judge LLM performance metrics, noting that even in highly controlled environments, state-of-the-art models still lag at least 9% behind human alignment. * Recommend using smaller models also such as Llama 3.1 8B and Mistral 7B for objective binary evaluation and ranking tasks, respectively. * Advocate for prompt engineering to optimize human alignment first and then use it on Judge LLMs, while cautioning against excessive instruction complexity. **2. Future:** We’re planning to include more datasets like LLMBar that have an objective Q&A setup. **3. Generalizability:** These findings likely extend to diverse datasets and tasks. If the scott’s agreement is 88% even in this controlled setup, then a more complex setup suggests potentially greater challenges in more complex scenarios, such as open-ended questions with or without gold standards. **4. Human alignment thresholds vary across datasets:** Previous literature has noted ~80-85% human alignment for open-ended tasks [1][2] and ~90-96% for objective evaluations [3]. Our work achieves a 98% agreement percentage across 600 questions. During experimentation, we noticed the tradeoff to accepting lower agreement percentage is that it limits the potential optimization of Judge LLMs due to inherent dataset ambiguity. So, Judge LLM performance reaches its peak when the dataset is least ambiguous, and their limitations are inherently tied to the level of uncertainty present in the data they are evaluating. **References -** [1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023) [2] Strong and weak alignment of large language models with human values (Kharmassi et al., Nature 2024) [3] Evaluating Large Language Models at Evaluating Instruction Following (Zeng et al., ICLR 2024)

We appreciate the reviewer's assessment. To address the identified shortcomings, we have prepared a point-by-point response to each of the concerns mentioned. To add context, contributions to the community is talked first. **1. Contributions to the community:** These are some of the contributions that we’d like to highlight that has a direct impact on the research community and industry - * The fact that the judges and humans differ in agreement in a comparatively simpler setup suggests that they might struggle in more complex setups such as open-ended questions with or without gold references. The limitations of our work are discussed in-depth in Appendix A. * Large Judge LLMs show strong alignment with humans, but smaller models like Llama 3.1 8B and Mistral 7B also demonstrate impressive performance in ranking and accuracy. * Our research quantitatively reveals Judge LLMs' tendency towards leniency (Section 5.4). We also found that excessive instructions can confuse LLMs (Section 5.2), resulting in lenient responses during binary or spectrum evaluations. **2. Goal of the paper and choice of TriviaQA:** We sought to establish an upper bound on Judge LLM performance in ideal conditions. This approach provides a clear benchmark for the community. We created an unambiguous, "sterile" setup to isolate core capabilities and assess maximum potential without confounding variables. In our preliminary study, when we started with Natural Questions, we couldn’t reach human alignment of ~90% because of ambiguous questions such as “Where did the term liberal arts come from ?”. The MT-Bench paper [1], which also concluded GPT-4 as leading Judge LLM, also observed that human alignment was ~81%. In this scenario, it’s extremely hard to optimize Judge LLMs when the dataset itself is so ambiguous that humans can’t agree to a correct answer. To address this, we chose TriviaQA dataset which contains factual questions with multiple references which eliminates the possibility of error in judgment being because of ambiguity in the questions/references. Consequently, we achieved 98% percentage agreement and 96% average alignment (Scott’s Pi) on a sample size of 600 between 3 annotators, demonstrating the reliability of our evaluation setup even further (discussed more in Appendix I). **3. False positive rate of Judge LLMs:** Thank you for bringing this to our attention! We’ll fix the text in paper to reflect that well aligned models tend to produce more false positives than negatives. This is further backed by Section 5.4 where we discuss that Judge LLMs tend to be more lenient. Having read the rebuttal, would you be willing to adjust your evaluation score? Thank you! **References -** [1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023)

We are grateful for the reviewer's evaluation. In response to the concerns raised about weaknesses, we have provided an itemized reply addressing each point. **1. Evaluation relies solely on TriviaQA:** We intentionally chose TriviaQA for our study, as it provides a well-defined benchmark to assess LLM judging capabilities while minimizing potential ambiguities in the evaluation process. TriviaQA's multiple references per question provide a more robust ground truth, and our inter-annotator agreement (98% on 600 instances) surpasses other datasets like LLMBar's (94% on 80 instances). Our larger sample size offers greater statistical power and sets a clear upper bound for human alignment in factual Q&A tasks. While additional datasets may be valuable for future work, our current methodology provides a focused, reliable assessment of LLM judging capability, supported by high inter-annotator agreement and a substantial sample size. **2. Lack of scope:** * Integrating code writing and evaluation would be an excellent extension of this work. However, smaller models currently struggle to generate high quality code comparable to larger SOTA models. Expanding our infrastructure for code execution requires additional funding and additional work to explore future ambiguities and iterate on new human guidelines. Our current setup is valuable as it accurately simulates how Judge LLMs are used in industry, providing relevant insights into their real-world performance. * We’ll be happy to move issues with Kappa in the main paper as we also believe that this lesser known fact will help practitioners adopt Scott’s pi metric over Cohen’s kappa. Having read the rebuttal, would you be willing to adjust your evaluation score? Thank you!

We sincerely appreciate the reviewer's thorough assessment. We address the reviewer’s concerns about the weaknesses of the paper below. **1. Limited Scope:** Our focus on TriviaQA is deliberate and advantageous, establishing a clear baseline for LLM judging ability with minimal ambiguity. TriviaQA's multiple references per question provide a more robust ground truth, and our inter-annotator agreement (98% on 600 instances) surpasses LLMBar's (94% on 80 instances). Our larger sample size offers greater statistical power and scope, while setting a clear upper bound for human alignment in factual Q&A tasks. While additional datasets may be valuable for future work, our current methodology provides a focused and reliable assessment of LLM judging capability in academic and industrial research. **2. Over strict evaluation setup:** Our use of exact wording matches over semantic evaluation was intentional, reflecting real-world industry applications of Judge LLMs. In our preliminary research, we saw that semantic evaluation may lead to higher Judge scores but reduced human alignment. This methodology also allows for more consistent comparisons across different models and tasks, establishing a solid baseline for future research in this area. **3. Further analysis:** While a power analysis on the evaluation sample would be ideal, we’ve mentioned high statistical reliability of sample in Appendix I and performed error analysis on the evaluation sample, categorizing common errors into 5 groups (Section 5.2). Even the best Judge LLMs frequently provided completely incorrect or under-specified (partially incorrect) responses. Examples using GPT-4 and gold references are provided below. After implementing guidelines (Appendix G), below answers were identified as under-specified. Outside of these examples, most of GPT-4's errors were due to incorrect entities or factual inaccuracies. > **Q -** Which language - originally a form of Anglo-Chinese jargon - was used by traders and businessmen on the China coast? > > **GPT4 -** Pidgin > > **Gold -** Pidgin English, English based Pidgins, English Pidgin > **Q -** In the USA, the Tav HaYosher is a certification mark offered to establishments that do what? > > **GPT4 -** Serve Kosher food > > **Gold -** Serve Kosher food and treat their workers fairly In light of the rebuttal comments, would you be willing to update your score? Thank you!

We are grateful to the reviewer for their comprehensive evaluation and specific feedback. We address each point raised in the review individually. **1. Lack of generalizability:** The reviewer points out that our work is a reference-based QA evaluation task which might not generalize to open-ended QA without the references. We have tested it on other datasets such as Natural Questions, and we have found that due to ambiguous questions (example question from NQ dataset - “Where did the term liberal arts come from ?”), we achieved less than 90% alignment scores among human annotators and Human - Judge LLM which can be attributed to the ambiguity in the question and references. Our choice of setup aimed to minimize ambiguity and establish an upper bound for human-LLM alignment in judging responses. By using TriviaQA's factual questions with multiple references, we reduced the potential for errors due to ambiguous questions or references. The observed differences in agreement between judges and humans in this sterile setup suggest that more complex scenarios, like open-ended questions without gold references, may present even greater challenges. Research conducted in the MT-Bench paper [1] also identified GPT-4 as the top-performing Judge LLM, but noted that its human alignment on the LLM evaluations was limited to ~81%. In this scenario, it’s then extremely hard to optimize Judge LLMs, when the dataset itself is so ambiguous that humans can’t agree to a correct answer. While LLMBar [2] achieved 94% agreement rate on a sample size of 80 instances between 2 annotators, we achieved 98% percentage agreement and 96% average alignment (Scott’s Pi) on a sample size of 600 between 3 annotators, which is a more reliable setup for our evaluation of Judge LLMs. **2. Novel contribution of Judge LLM correlation with human eval:** To our knowledge, this is the first study that is using Scott's pi to assess agreement between Judge LLMs across a wide range of model sizes (2B to GPT-4) and families. While existing work appreciates the performance of SOTA Judge LLMs, our work uniquely reveals the surprisingly strong performance of smaller models like Llama3.1-8B and Mistral-7B compared to larger ones. These findings could encourage more research into compact models for various applications, benefiting the research community. **3. Scott’s pi claims are not well supported:** Scott's π is preferred not only for its broader numerical scope but primarily for its ability to assess inter-judge agreement beyond coincidence. This offers a more sophisticated analysis of judges' evaluation performance [Appendix B] and has been explored in literature as well [3]. By introducing Scott's π, we aim to promote a more accurate measure of inter-rater reliability across various fields. The statistical reliability of Scott's π has been explored in literature [4] [5], and Appendix I discusses the reliability of evaluation samples through consistent Scott's π scores. **4. True Negative rate of Judge LLMs:** We appreciate your observation! Indeed, EM/Contains Judge LLMs show higher recall than precision as expected. We initially included these to demonstrate lexical metrics' limitations in assessing semantic richness, despite their alignment with some Judge LLMs. We'll update the paper to mention this in the text and remove EM/Contains from the Precision/Recall graph for clarity. **5. Information about annotators and recruitment criteria:** Preliminary research involved iterative refinement of human annotation guidelines [Appendix G] to ensure consistency and reproducibility across annotators with general English semantic knowledge. CS graduate students served as annotators for this experiment. In light of the rebuttal comments, would you be willing to update your score? Thank you! **References -** [1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023) [2] Evaluating Large Language Models at Evaluating Instruction Following (Zeng et al., ICRL 2024) [3] Interrater reliability estimators tested against true interrater reliabilities (Zhao, PMCMed Central, 2022) [4] Answering the Call for a Standard Reliability Measure for Coding Data (Hayes et al., Communication Methods and Measures 2007) [5] A Critical Analysis of Inter-Coder Reliability Methods in Information Systems Research (Nili et al., ACIS 2017)