Summary
This paper proposes a meta-reasoning benchmark for evaluating the solutions generated by a large-language model (LLM) to shift the focus more to process-based evaluation of an LLM's reasoning abilities rather than outcome-based evaluation.
*Evaluation*:
On a variety of question-solution pairs, the model is asked to score its own generated solution for correctness, to identify the first erroneous step, and if these are correctly identified, the reasoning of it solution is evaluated by GPT-4. These three are used to compute three metrics respectively-- the MCC for a binary classification of the solution correctness, the ACC_step to judge the number of correctly predicted first-error steps as compared to the total number of incorrect solutions, and the ACC_reason to judge the number of correctly predicted first-error steps and error-reasons as compared to the total number of incorrect solutions. These three are combined to form an MR-score that is used to compare different models.
*Dataset*:
The question-solution pairs (~6k in all) themselves are formed such that the questions are sampled from MMLU (arithmetic reasoning), LogiQA(logical reasoning), and MHPP (code-based reasoning) and the solutions are generated via chain-of-thought prompting by GPT-3.5-Turbo-0125, Claude-2, and Mistral-Medium. The annotations for these question-solution pairs are annotated by human annotators-- these annotations of solution correctness, first error-step, and the error-reason are used for evaluation.
Strengths
- The authors highlight the importance of more carefully evaluating the solutions generated by LLMs to complex problems rather than just comparing their accuracy which is a notable contribution to the body of work of improving evaluation of LLMs.
- The annotation process followed for experimental evaluation is thorough and multi-step for quality assurance
- The authors raise an interesting question about problem solving dynamics, which can be potentially explored further in future work on LLMs
Weaknesses
These questions/remarks might also have arisen due to my lack of proper understanding, so I am willing to increase my score if these can be clarified:
- My main issue is with the mixing of reasoning based and accuracy based evaluation into a singular score. What is the reasoning behind combining the three metrics (MCC, ACC_step, and ACC_reason) into a single score (MR-Score)? Does this not affect the motivating insight that models might have a high MCC but a super low ACC_step and an even lower ACC_reason? Is there somewhere the three metrics can be viewed individually also so it's clearer how the models vary along the three-- right now, it is not clear to me by looking at the MR-scores in Table 2 how the 15 models actually fare when it comes to identifying the correct error-reason, especially when the MR-scores for the models are in the same ballpark.
Writing:
- (pre-existing work): it's not clear to me what the real differentiating aspect of MR. Bean is vis-a-vis MR-GSM8K and MR-Math. MR-GSM8K and MR-Math also consider the first-error reason? So, is it that they are math-reasoning based, and MR. Bean now just combines multiple datasets to essentially follow the same evaluation pipeline? If so, this should be clearly mentioned. They don't just go a step further, that is the furthest step gone to, even considering this work (if that is indeed the case).
- The abstract makes it seem like the MR-score is a metric designed by the authors in this work, although it's already been proposed "*Through our designed metrics*" which is somewhat over-selling the work.
- (minor) typo: A.2 Negative Societal Impacts
- The connection between task difficulty and reasoning capability can be explored further-- I am assuming ACC_reason on more difficult tasks would be comparably worse than the MCC, which would be the actual assessment of reasoning capabilities. Also, from Fig 2, it seems that Logic based tasks are the most difficult, then wouldn't it be fair to include them in the comparison too or perform the comparison of task difficulty not just based on high-school/college, but also take into consideration how the model actually interprets the task difficulty based on MR-score?
Experiments:
- what is the main takeaway on self-refine prompting since it seems to be in contrast to existing observations? How does the shift from incorrect to correct predictions depend on the model family and task? Is it seen only across MR-Scores or also in the ACC_reason (which should probably be affected more by model size?)?
- Is the difficulty of logic-based reasoning also because they contain the longest questions? is there any impact of question length on the model's reasoning ability, if not solution length?
Questions
Some questions for clarification:
- I might have missed this, so in Table 2: is the meaning of k the number of shots of prompting the model gets? If so, it would help to mention that in the caption since k is not mentioned in the text anywhere else.
what should be the take-away on self-correction?
- Is the difference between the Self-Refine prompting (section 6.2) and Solution Correctness Prior (section 6.3) the absence/presence of of the ground truth (resp.)? How do the two compare with each other?
Limitations
- The proposed benchmark also relies on step-wise solutions to evaluate reasoning which reduces its applicability and novelty as compared to existing work.