Evaluating Large Language Models at Evaluating Instruction Following

Response to Reviewer 65gL [1/3]

Thank you for your constructive feedback! We agree that a strong capability in evaluating instruction-following is crucial for any LLM to serve as a reliable evaluator, and LLMBar bridges this research gap by investigating it. We believe that LLMBar, by offering high-quality and efficient meta-evaluation, will significantly contribute to the development of LLM evaluators. We will address your comments and suggestions in more detail below. **(1) The instructions used for dataset construction contain ambiguous words.** Thanks for raising this point! We acknowledge that some usage of words in the Natural set may seem ambiguous. It is intended as we sample from existing datasets to construct the Natural set and try to keep a real-world distribution. However, we made every effort to make sure that in the provided output pairs, one is objectively better than the other without ambiguity. Taking the “chicken in the library” instance in Appendix A.1 as an example, Output 1 is objectively better than Output 2 as Output 2 is not “imaginative” at all. Such objectivity in preferences is also reflected by the 90% human agreement rate of our Natural set. **(2) The authors use different prompting strategies for evaluation, but the dataset generation only uses plain prompts.** If you are suggesting using more sophisticated prompting for dataset generation, especially for the GPTInst subset (as it can change the distribution of instances’ instructions), we appreciate your suggestion and will explore this direction in future work to increase the diversity of our dataset. **(3) It is interesting yet important to discover how different input instructions enhanced by CoT or other prompting strategies affect the judgment of different LLMs** Reflecting on your proposed direction, we think this setup (of how instructions are like) is both practical and intriguing: the user's instruction includes "metrics" giving what a good or expected output should entail. As you mentioned, understanding how this kind of instructions influences LLM evaluators is important and interesting. Given the labor-intensive nature of manually creating such instructions, we propose utilizing LLMs to do that. This practice actually translates into the following prompting strategy: we first generate a list of metrics and then incorporate these metrics into the instruction to create an enhanced instruction $I_\text{enhanced}$. This enhanced instruction, along with the pair of outputs, is then fed into an LLM evaluator (we could use the basic **Vanilla*** prompting strategy here as the simplest version). We refer to this strategy as **Enhanced*** and evaluated it on LLMBar using GPT-4 and ChatGPT respectively. The results of the average accuracies are as follows: GPT-4: | Strategy | Natural | Neighbor | GPTInst | GPTOut | Manual | | :----------------: | :-----: | :------: | :-----: | :----: | :----: | | Vanilla* | 95.5 | 78.7 | 86.4 | 77.7 | 80.4 | | Metrics* | 93.0 | 83.2 | 89.7 | 73.4 | 81.5 | | **Enhanced*** | 92.0 | 81.7 | 88.6 | 71.3 | 77.2 | | Metrics+Reference* | 96.0 | 85.4 | 89.7 | 72.3 | 83.7 | ChatGPT: | Strategy | Natural | Neighbor | GPTInst | GPTOut | Manual | | :----------------: | :-----: | :------: | :-----: | :----: | :----: | | Vanilla* | 81.5 | 19.4 | 26.6 | 41.5 | 34.8 | | Metrics* | 81.5 | 28.4 | 35.9 | 41.5 | 43.5 | | **Enhanced*** | 81.5 | 33.2 | 29.3 | 42.6 | 43.5 | | Metrics+Reference* | 82.5 | 38.1 | 35.9 | 38.3 | 43.5 | The results indicate that **Enhanced*** significantly outperforms **Vanilla*** when using ChatGPT as the base LLM. This observation leads to two implications: (1) this strategy appears to effectively assist LLM evaluators, and (2) considering two meta-evaluation instances, even when their instructions convey similar intentions, variations in wording can significantly impact the performance of LLM evaluators. We believe these are both interesting directions for future research. Note: **Enhanced*** is quite similar to **Metrics***. The difference is that the LLM evaluator now regards the generated metrics as a part of the instance's instruction, instead of as a part of the evaluator's prompt (the prompt instructing LLM evaluators to do evaluation). This can be seen as a way to influence the distribution of instances' instructions. Therefore, it serves as both a strategy and an analysis experiment.

Response to Reviewer WsKe [1/2]

Thank you for the constructive feedback! We are encouraged that you recognize LLMBar as resolving a major missing piece in the high-quality, scalable meta-evaluation during the current surge of LLM evaluator usage. We appreciate your detailed comments and suggestions and will address them below in detail. **(1) Some confusion in how the evaluation set was generated.** Regarding “$O_1$ is generated by weaker models and $O_2$ by stronger models”, this is employed mainly for the collection of the Neighbor set. We had provided the details in footnote 5 (in the old version, now deleted) : 'We generate $O_1$ using an instruction-tuned LLaMA-7B, and $O_2$ is taken as the reference output from original datasets, which are generated by text-davinci-003 in Alpaca, by humans in OpenAssistant, and by ChatGPT in ShareGPT'. We appreciate your raised point and acknowledge that including this detail in a footnote may have caused confusion for readers. In the updated version, we have moved this detail into the main text of Section 2.2 for better clarity. **(2) Not sure what “manual filtering and modification” entail.** This term refers to the final stage in constructing both the Natural and Adversarial sets. In this process, each instance is carefully examined by the authors. If there is no objective preference between two outputs, or if the label is incorrect, we will then modify the instance accordingly or discard it if making such modifications is difficult. We have provided a more detailed explanation in the updated version of Section 2.1. For specific examples of this process, please check out Appendix A.1, A.2, and A.4. **(3) The human evaluators are co-authors of the papers, which can raise issues of objectivity compared with other crowd-sourced human evaluators.** We would like to clarify the procedure for conducting the human evaluation: For each instance, one author (annotator A) annotates the label, and another author (annotator B) annotates the instance again, without seeing the previous label. The annotator agreement rate is calculated as the percentage at which annotator A/B agree, which is also the human accuracy. To remain objective, the selected annotator B(s) have not seen the constructed data or annotator A’s labels before. We believe that this is a strength rather than a weakness. Our annotation task is objective by design and has very clear task definitions. The authors of the paper best understand the task instructions, and therefore, are best positioned to provide high-quality annotations. On the other hand, crowdworkers recruited through platforms like MTurk are more likely to introduce noise, such as misunderstanding task instructions. Due to this exact reason, several works in the past refer to author-annotated data as the “expert” test set, contrasting it with crowdsourced data that is assumed to be of lower quality but easier to collect at scale (Goyal et al., 2022). If the goal was to collect a diverse set of instructions or queries that reflect real-world usage distributions, then recruiting a diverse set of crowdworkers would be critical. However, in our setting, letting authors do the annotation and human evaluation is the best approach.

Response to Reviewer WsKe [2/2]

**(4) I am wondering whether it is a fair comparison to other benchmarks if the instructions are qualitatively different. For example, it is relatively easy to achieve high inter-rater reliability if the response is factual information rather than subjective.** We agree that LLMBar is qualitatively different from other benchmarks, and is designed to focus on one particular aspect of evaluation, i.e., instruction-following. We believe that it fills a critical gap in prior work. While studying previous meta-evaluation benchmarks (FairEval, LLMEval^2, MT-Bench), we observed that they cannot distinguish between different evaluators (see Figure 5) and fail to provide recommendations into which LLM evaluators are strongest. We hypothesize that one reason was that prior benchmarks cannot disentangle the effects of subjectivity and noise in their annotations, i.e., is the disagreement between annotators due to annotation errors/noise, or is it because the choice is inherently subjective? Therefore, they end up scoring most evaluators similarly. Our LLMBar benchmark addresses this gap in prior work by providing an **objective** (and also challenging) benchmark for comparing different evaluators. We intended to show such differences via our experiments and analysis. We show that + LLMBar is more objective and the annotation quality is higher, hence it achieves higher human agreement as shown in Section 4.2. The objectivity and high-quality annotation make our benchmark more reliable and indicative. + LLMBar is more challenging and distinguishes different evaluators better. This is demonstrated by Figure 5, where different evaluators perform similarly on other benchmarks but distinctly on LLMBar. We argue that our experiments deliver a fair comparison, as the goal is to show how challenging, objective, and high-quality our benchmark is compared to other benchmarks via the experiments. In the future, we hope that LLMBar can serve as a “stress test” to compare different evaluators, possibly in addition to other aspects (other than instruction following) that practitioners want to evaluate. **(5) It would be ideal to have another benchmark with comparable features (e.g., objectiveness and similar inter-rater agreement)** We believe that LLMBar is the **first** meta-evaluation benchmark that ensures objectivity and achieves an inter-annotator agreement of more than 90%. We focused on instruction following due to its importance in using LLMs. We hope that future work will build on the ideas in this work and introduce other objective benchmarks, possibly relating to other aspects apart from instruction following. **(6) Please state the metrics used for calculating inter-rater reliability, e.g. Cohen Kappa etc. making sure it is consistent with other papers to be comparable.** Our human agreement/accuracy is calculated as the percentage at which the two annotators agree with each other. In Section 4.2, we refer to the average human accuracy in FairEval and the agreement in MT-Bench is the average agreement among each pair of two annotators, so our calculation is mathematically equivalent to those of other papers we compared. **(7) Typo in the caption of Figure 5** Thanks for pointing it out! We have fixed it in the updated version. ## Reference Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. SNaC: Coherence Error Detection for Narrative Summarization. In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 444–463, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Reminder of the Discussion Period Deadline

Dear Reviewer WsKe, We hope this message finds you well. As we're approaching the last day of the reviewer-author discussion period, we wanted to gently follow up regarding our previous response to your review. We value your input highly and would greatly appreciate it if you could let us know whether our rebuttal has addressed your initial concerns, which were primarily clarification questions. Your feedback is crucial for us, and we remain open to further discussion if needed. Thank you once again for your insightful review and the time you have invested in it. Best regards, Authors 7039

Response to Reviewer kxW7

Thank you for the constructive feedback! As LLM evaluators become more commonly adopted, we are encouraged by your appreciation of the contributions that LLMBar and our prompting strategies provide to the research community. We believe LLMBar will contribute to standardized meta-evaluation for future LLM evaluators. To address your concern **Are LLMs likely to sample non-preferred output for instructions in the adversarial pool? Could you provide the distribution of generating preferred and non-preferred outputs for a few LLMs?**: + We randomly sampled 50 instructions from Adversarial and found that ChatGPT (`gpt-3.5-turbo-0613`), LLaMA-2-70B-Chat, and PaLM2 (`text-bison-001`) can faithfully follow 90%, 86%, and 84% of them respectively by human evaluation. + Even though the LLM can generate instruction-following outputs, it does not mean the LLM can discern instruction-following outputs. This is shown in our results and echoed by West et al., 2023, which shows that being able to generate does not entail being able to understand. In this work, we focus on studying whether the LLM can distinguish instruction-following outputs from dispreferred outputs. The adversarial instances are constructed such that it is challenging for LLM evaluators to identify the instruction-following outputs, rather than being deliberately hard to generate based on the instruction. We have added this clarification in Section 2.2 of our updated version. Due to the page limit, we plan to discuss this point to provide more insights in our next version. + Even GPT-4 is likely to generate outputs not following simple instructions (Wu et al., 2023; Li et al., 2023). This is a key motivation for our focus on evaluating whether LLM evaluators can detect instruction-following outputs, an effort that can facilitate future development of instruction-following models. For example, in the Neighbor set, there is an instance asking for **methods of learning** search engine optimization (SEO). The dispreferred output deviates by providing **specific ways of doing SEO**. We observed that GPT-4, when prompted with this instruction, tends to produce outputs similar to the dispreferred one. In response to **whether the findings will improve LLM evaluators' ability**: Though LLMBar’s Adversarial set has a distinct distribution, we believe that an improvement on it indicates a more reliable LLM evaluator in the real-world distribution – being able to distinguish instruction following from other superficial appeal should be a basic requirement for LLM evaluators, and failing to do so makes the evaluator misleading. However, we acknowledge the distribution shift and that the Adversarial set is more like a “stress test”, hence we also provide the Natural set as a reference. We see that the performance trends on the Natural set generally agree with that on the Adversarial set but with a smaller margin. ## Reference Wu, Zhaofeng, et al. "Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks." *arXiv preprint arXiv:2307.02477* (2023). Li, Shiyang, et al. "Instruction-following evaluation through verbalizer manipulation." *arXiv preprint arXiv:2307.10558* (2023). West, Peter, et al. "The Generative AI Paradox:" What It Can Create, It May Not Understand"." *arXiv preprint arXiv:2311.00059* (2023).