Critique Ability of Large Language Models

Thank you for your thorough and detailed feedback, reviewer qpMo. We are grateful for the time you've invested in reviewing our paper. We regret that certain aspects of our paper may not have been as clear as intended, which seems to have led to some misunderstandings. Rest assured, we will endeavor to clarify these points as follows. 1. We've addressed this in our "common response to all" and invite you to review it for more details. Regarding your use of the word "claim", which means *“state or assert that something is the case, typically without providing evidence or proof”*[1], we understand it implies we state to evaluate critique ability while actually not. We welcome specific insights on why you believe we fail to do so. We've included tasks including QA&Classification, reasoning, coding in the paper, and additional preliminary numbers for generation tasks in the rebuttal. However, we're open to suggestions on other NLP tasks or domains we might have missed and are eager to incorporate them in future work. Lastly, there is no dataset regarding common-sense in our paper. 2. We understand the workload of reviewing tasks can be heavy, leading to skipping some details in the paper. We are more than willing to guide you through the critical sections of our paper. Specifically, in Section 3.2, spanning pages 3 to 6, we explore various factors that could potentially *“influence evaluation results”*. This includes the quality of queries and responses, the impact of model sizes, and model certainty. Moreover, for your convenience, we draw your attention to Sections 4.2 and 4.3, on pages 7 to 9, where we delve into how model size and certainty play a pivotal role in evaluation results. Those discussions and empirical observations provide valuable insights for us on how to construct a robust evaluation benchmark — which is the main focus of this paper. 3. We made every effort to comprehend the question but failed. What exactly does *"why self-critique could be better"* mean? Are you asking whether self-critique should be superior to normal critique, or if it should be improved beyond what we presented in the paper? We never state anything close to *“self-critique should be better (than something)”*. Assuming we understand the basis of such a question, then what is the meaning of *"its (the reason's) influence on critique capabilities”*? By definition, self-critique is a special form of critique. Thus, the question *"(a reason) why self-critique could be better (than something unspecified) and its influence on critique"*, remains unclear to us. We recommend that the reviewer refine their question to make its intention and meaning clearer to the audience. 4. We regret that the paper seemed challenging to comprehend. We acknowledge that if certain sections, particularly those between pages 3 and 9 based on your Question 2, were skipped, the flow of content might indeed appear disjointed. Our sincere apologies for any bad experience this might have caused. We are open to constructive feedback and would greatly appreciate any specific suggestions you might have to enhance the clarity and coherence of our work. 5. In the context of LLM, where RLHF [2] being a very common and basic approach for LLM tuning, “policy model” typically means the LLM that takes inputs (states) and makes actions (outputs) for the base task. “Critic” means *“a person who expresses an unfavorable opinion of something”*[1], we simply use its literal English meaning to indicate a model providing critiques. 6. Due to the extensive content and details in the paper, we have chosen to include only a brief discussion of related work in Section 2. We thank the reviewer for highlighting this issue and will consider adding more detailed literature in the appendix. 7. This is incorrect. (1) The dataset is ready but currently under legal and compliance review to ensure safety and prevent the leakage of sensitive information. (2) Our institution has very strict rules regarding data publication, requiring all data releases to go through official channels. To maintain the anonymity of the submission, it is impossible to show the data during the review period. This is a common practice in large and responsible institutions, and we do not anticipate criticism for this decision. Answers to questions: 1. PaLM-2 tech report (Google et al., 2023) demonstrates that such choices of k could guarantee an acceptable rate of correct answers for questions in those datasets. 2. Due to the length limitations imposed by ICLR, we cannot include all details in the main body of the paper, particularly for the prompt template, which is quite extensive. As mentioned at the beginning of Section 4, all detailed settings, including the construction of the prompt, are described in Appendix F. Specifically for Critic-GSM8K, all the details are available in Appendix F.1. [1] Oxford Languages. https://languages.oup.com/google-dictionary-en/ [2] https://arxiv.org/pdf/2009.01325.pdf

- In the three references the reviewer mentioned, one [3] became publicly available **after** our ICLR submission, and another [2] even **did not exist on the public internet** when we received the review. We are open to including further discussion on these studies in future revisions. However, for this ICLR submission, with the deadline being **September 28**, we believe it would be **UNFAIR** to be criticized for not discussing papers submitted to arXiv in **October** or **November**. Additionally, the main focus of our paper is the comprehensive evaluation of the critique ability of LLMs, as detailed in Sections 3 and 4 including extensive discussions on scalability, generalizability, and quality. The self-check baseline, as emphasized in the paper, serves merely as a simple example demonstrating the potential use of critique ability. Although this method surpasses some prior approaches like [1], we actually do not consider it a major contribution to our work. - We've addressed this in our "common response to all" and invite the reviewer to check it for more details. - The purpose of this paper is to introduce a standard framework for assessing the critique ability of LLMs. The questions raised here fall outside the scope of this paper. Nevertheless, we do discuss why models underperform on certain tasks and propose potential solutions. - In Section 4.1, regarding Critic-HumanEval, we note: *"This is somewhat anticipated, as evaluating the correctness of a code snippet without execution is often challenging even for expert software engineers. It is likely to gain a notable improvement when augmented by a code interpreter tool."* - Furthermore, in Section 4.2 on Critic-TruthfulQA, we note: *"We hypothesize the disparity is due to the underlying reasons of a model answering incorrectly to queries. For TruthfulQA, the wrong answers largely stem from false beliefs or misconceptions in models, which would also lead to critique failures."* - This question is far beyond the scope of this paper. None of the references mentioned in the first question of this comment address issues like human oversight. In fact, we did discuss this topic in the last paragraph of Section 2. Please refer to the paper for more details.

Thanks for the time and effort! Please find our responses below. ## Binary metric only offers high-level granularity We are aware of the limitations of a binary (correct/incorrect) metric regarding granularity. However, this is actually an intentional choice. 1. Firstly, regarding motivation: This approach is a **trade-off** for better **generalizability**. Although we could design more fine-grained metrics for evaluation, such as distinguishing between calculation errors and reasoning errors in math tasks, these metrics are not as generalizable to other tasks. For example, in coding tasks, people may care more about whether there are syntax errors, compile errors, or runtime errors. Considering the limited studies on the critique ability of LLMs in the community, we prioritize broader generalizability across various domains and task types rather than delving deeper into one specific task. 2. Secondly, in terms of practice: Despite its limitations, a binary metric can still provide insights into how well a model critiques. In the second paragraph of Section 4, we note, *"In evaluation, we focus solely on the accuracy of this final judgment, disregarding the correctness of the intermediate analysis, as empirical evidence has shown a strong correlation between the accuracy of intermediate chain-of-thought and the final answer (Wei et al., 2022b; Lewkowycz et al., 2022; Fu et al., 2023a)."* ## “even if the model critique doesn’t make sense to a human, as long as it improves subsequent prediction accuracy, it’s a good critique.” We **strongly disagree** with this statement due to its potentially **dangerous** implications. For instance, consider the task of determining patients' depression levels based on their therapy session transcripts. A model might incorrectly assess a depressed or anxious minority patient as having a low level of depression. If a critique model then suggests, “The patient mentioned they are a minority. All minorities are weak and vulnerable, so they should be classified as having high-level depression,” this could make the downstream task to correct its prediction. Based on the reviewer’s statement, the subsequent prediction accuracy increased so it’s a good critique. However, the critique here is not only illogical to humans but also **biased** and **harmful**. This is a realistic task and it's a fact that minority groups often exhibit higher levels of depression in psychological studies, typically due to exposure to discrimination and unfair treatment, especially during childhood [1]. Therefore, the critique mentioned above is not only inaccurate but also detrimental, despite its potential to increase downstream accuracy. Evaluating the critique capabilities of LLMs is crucial. Downstream accuracy is not the sole indicator of effectiveness. An accurate, precise, and reasonable critique itself is equally important. Our current benchmark represents an initial step in thoroughly assessing the critique abilities of LLMs. While it is not yet perfect, we recognize the significance of this task. [1] https://www.nature.com/articles/s44159-023-00241-5