Summary
The paper introduces ReaLMistake, which the authors claim is the first error detection benchmarking suite consisting of objective, realistic and diverse errors made by LLMs. ReaLMistake contains three tasks that introduce objectively assessable errors in four categories (reasoning correctness, instruction-following, context-faithfulness, and parameterised knowledge), eliciting naturally observed and diverse errors in responses of GPT-3 and Llama 270B annotated by experts on three tasks three tasks (Math Word Problem Generation, Fine-grained Fact Verification, and Answerability Classification). The benchmark includes error annotations (binary error label, error categories, and human explanation about errors) on responses from GPT-4 and Llama 2 70B on the three tasks. The authors show that top LLMs like GPT-4 detect errors made LLMs at very low recall, and all LLM-based error detectors perform much worse than humans, that explanations by LLM-based error detectors lack reliability and LLM-based error detection is sensitive to small changes in prompts but remains challenging to improve, and finally, popular approaches to improve LLMs, including self-consistency and majority vote, do not improve the error detection performances. The authors promise to make the code and data available on acceptance.
Overall the quality of the paper is quite high in terms of the detail of the description of the dataset, though this is somewhat in part due to the lengthy Appendices making very clear what the prompts and responses were. ReaLMistake undoubtedly could be part of a good resource for evaluation for LLM error detection, though it is limited in scope, as partially admitted to by the authors. The presentation overall is quite strong and clear, though see below for some queries about it which should be addressed. It seems more incremental research than highly novel, as the authors define the benchmarking dataset as an intersection of what has gone before, however the paper claims ReaLMistake is novel in focusing on errors that are unambiguously judged as such by human annotators (though see concerns on this claim below). The approach is quite principled scientifically, though I would dispute the claim of unambiguously annotated error data as being essential for good error detection benchmarking for LLMs, and perhaps if accepted, the authors can reduce the strength of that claim and describe alternatives where multiple annotator’s judgments are included (see below).
Reasons to accept
There is a useful data contribution of the 900 instances of error annotations by experts, including human explanations for the judgment of an error on responses from GPT-4-0613. This is potentially very useful for certain kinds of error detection benchmarking and is challenging, as shown by the authors who show that near state-of-the-art models such as GPT-4x exhibit a very high error rate in detecting errors in certain tasks with the prompts designed by the authors.
The statistics of the ReaLMistake benchmark are clear and show a good range of errors in the different dimensions defined by the authors.
The Appendices are very useful as a clear exemplification of prompts and further results, which when included, mean the reproducibility of the paper is quite high from the text alone.
Overall the paper is well-written, and fairly easy for readers to understand the method and approach employed.
Reasons to reject
My main reservation about the paper, which could be corrected if claims were slightly toned down, is in terms of the motivation of the form of the benchmark: the authors claim they are against human subjective ratings of errors in a response, or at least instances which are not universally agreed upon for their label (e.g. “We cannot evaluate error detectors if humans cannot determine whether LLM responses include errors”). However, this claim is too strong, and it is not clear how the inclusion of multiple (potentially differing) annotations would make things worse for a benchmark – they are certainly a challenge to be solved going forward, but rigorous inter-annotator agreement studies in creating these benchmark datasets could be a way to deal with ambiguous/non-unanimous labelling - in fact it could even be a good idea to include the differing annotations in the benchmark. The challenge would then shift away from limiting a dataset to unambiguous error instances only to developing systems that give similar confidence levels to the level of inter-annotator agreement. Even in some cases there are mathematics-based prompts, there is still room for ambiguity in the interpretation of verbal prompts.
It is not clear whether the explanations were agreed upon between the annotators.
It is not made clear how novel the benchmark is as Table 1 suggests ‘MT-Bench, PandaLM, LLMEvalˆ2 Multiple NLP Tasks’ has more criteria and in multiple domains.
Questions to authors
I would reduce the strength of the claims around ambiguity and frame your dataset as collecting error judgements which are low in ambiguity (as illustrated by the 95% F1 score of agreement), rather than ‘objective’. This is scientifically too strong.
The claim of it being a diverse set of errors might also be a bit strong, as it is dependent on the diversity and difficulty of the original tasks – I would reduce this and discuss that there is more work to be done to enlarge the scope of error type.
As per the above, it is not clear how novel the approach is, given Table 1’s column ticks – please further clarify how it is going further than the top row’s system in the text, particularly if you lessen the claim that human inter-subjective disagreement is a bad thing for annotation overall.
Please calculate and report Cohen’s kappa for the agreement score, which is similar to F1 but more comparable to other inter-annotator agreement scores.
There are some errors in phrasing throughout which require a proof-read - e.g.
- Use ‘LLM-based’ (rather than LLMs-based)
- Use determiners in places like ‘We introduce ReaLMistake benchmark’ –> ‘We introduce the ReaLMistake benchmark’