Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist

Response to Reviewer n8tf

Thank you for reviewing our work. Below, we will address your concerns point by point: > The evaluation methods proposed in the paper may not be challenging enough for the most advanced models currently available. **The goal of MathCheck is to provide a more comprehensive evaluation of mathematical reasoning abilities at a given difficulty level, rather than simply making math problems harder to stump the model**. To this end, we design a test checklist from both reasoning robustness and task generalization. It allows users to better evaluate the math reasoning ability and conducts fine-grained analysis. Besides, it is a plug-and-play testing framework suitable for mathematical problems of different difficulty. We construct MathCheck-GSM for two motivations: (1) validating the effectiveness of MathCheck paradigm. (2) determining whether LLMs are capable of reasoning at the grade school level while most of them achieve above 85% on GSM8k before. **Through our experiments, we observe significant performance drops in many LLMs, which validates the effectiveness of our paradigm and reveals that some LLMs rely on memorization to solve GSM8k problems.** Meanwhile, we also want to evaluate models on MathCheck with multimodal math problems. To address this, we developed **MathCheck-Geo comprising high-school geometry problems, on which GPT-4o achieves only 65.3% and we successfully identified weaknesses (robustness or task generalization) in some MLLMs.** > Not find any new insights proposed by the evaluation method for the improvement of existing models. Evaluation is of significant importance to assist the research and development of LLMs. Our benchmark offers several insights w.r.t. analyzing and improving the mathematical reasoning abilities of LLMs in the following aspects: 1. **Fine-grained analysis**: MathCheck offers various robustness variants and task types, enabling users conduct fine-grained analysis on their models and target weaknesses such as robustness. It helps further improve the general math reasoning abilities of models 2. **Identification of pre-training generalization**: Most base models exhibit strong reasoning consistency on MathCheck, indicating that the mathematical abilities learned from pretraining are generalized and robust. 3. **Insights on fine-tuning and data augmentation**: Figure 5 implies that finetuning solely on massive problem-solving data is not the right direction to improve general math reasoning abilities, highlighting the need for more diverse and high-quality SFT data. > The multiple aspects proposed seem insufficient: if one model performs well in all the proposed aspects, does it truly have reasoning abilities? In order to make the problem variants comprehensive, we propose to **construct test samples from reasoning robustness and task generalization**. In reasoning robustness, we consider Problem Understanding, Irrelevant Disturbance, and Scenario Understanding. In task generalization, we consider answerability detection(Answerable Judging) , answer correctness judgment (Outcome Judging) , process-error identifying (Process Judging). **These problem variants comprehensively test multiple reasoning skills in mathematical reasoning**. Compared to previous evaluation paradigms, we make a big step forward: MathCheck is **more closer** mathematical intelligence, which we have validated in Section 3.4 through correlation with Private Data and Compression Efficiency. > How to ensure the diversity across different question groups? For example, with different seed questions, LLMs may tend to include similar irrelevant information when generating the “irrelevant disturbance” variety, leading to a less comprehensive evaluation. **We have considered diversity during the generation process.** - When generating Irrelevant Disturbance question variants, LLM creates distractors related to the original question's topic, ensuring the irrelevant information is unique. For instance, in the original question: _”A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?”_ its variant is: _“A tailor is crafting a luxurious robe. The design requires 2 bolts of blue fiber and half that amount of white fiber. **To add grandeur, the tailor also considered using 3 bolts of golden thread from the sun's rays, but eventually decided it would be too gaudy for the ceremony.** How many bolts in total are needed for the robe, disregarding the golden thread?”_ We can see that the distractor is topic-related. - Furthermore, when generating binary-classification tasks (Answerable Judging and Outcome Judging), we created both a positive and a negative sample for each problem. - For Process Judging task, we instruct the LLMs to generate error of given a random step rather than fixed step, ensuring diversity in the generated error.

Further response to Reviewer n8tf

> In Sec. 3.1, the number of samples in MATHCHECK-GSM is reported as 3096. Could you clarify this discrepancy? In each group, since answerable judging and outcome judging are binary-classification tasks, we include two different labels in these units for fair evaluation. For example, in each answerable-judging unit, we have 2 samples (Answerable and Unanswerable). This distribution ensures that our data is balanced and diverse. **Therefore, a seed problem is expanded to 4+4$\times\$2+4$\times\$2+4 =24 problems. For the 129 style groups in MATHCHECK-GSM, the total number of samples is 129$\times\$24=3096.** In the original version, we have provided relevant descriptions in Appendix C.1 and Table 5, please check. Thanks for your review again. We sincerely hope that our responses can address your concerns and contribute to a better evaluation on our work.

Response to Reviewer n8tf

Dear Reviewer, We sincerely appreciate your feedback and effort. During the rebuttal process, we have earnestly addressed your concerns and questions with detailed, point-by-point replies, including: 1. The number of samples in MathCheck-GSM. 2. The effectiveness of MathCheck. 3. Insights for improving general mathematical reasoning. 4. The diversity of generated data. We sincerely hope that our responses could satisfactorily address your questions and concerns. If you have any further inquiries or require additional clarification, please do not hesitate to let us know; we remain available and willing to address any additional questions or provide further modifications as needed. Once again, thank you for your valuable input! Best wishes, Authors of Paper 7051

Response to Reviewer GGUq

Thanks for your time and valuable comments. Below, we will address your concerns and suggestions point by point: > Citing functional variations benchmarks. Indeed, the Functional Variations Benchmarks are highly relevant to our work. We have now cited two studies on Functional Variations Benchmarks in the Related Work section [1][2]. Reference: [1] Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. (Srivastava et al., Arxiv 2024) [2] Putnam-axiom: A functional and static benchmark for measuring higher level mathematical reasoning. (Aryan Gulati et al., NeurIPS 2024 MATH Workshop) > Performance of MathCheck-GSM. **The goal of MathCheck is to provide a more comprehensive of mathematical reasoning abilities at a given difficulty level, rather than simply making math problems harder to stump the model.** Base on this, we design a test checklist from reasoning robustness and task generalization. It allows users to better evaluate the math reasoning ability and conducts fine-grained analysis. Besides, it is a plug-and-play testing framework suitable for mathematical problems of different difficulty. We construct MathCheck-GSM for two motivations: **(1) validate the effectiveness of MathCheck paradigm. (2) determine whether LLMs are capable of reasoning at the grade school level while most of them achieve above 85% on GSM8k before.** Through our experiments, we observe significant performance drops in many LLMs, which validates the effectiveness of our paradigm and reveals that some LLMs rely on memorization to solve GSM8k problems. Meanwhile, we also want to evaluate models on MathCheck with multimodal math problems. To address this, we developed MathCheck-Geo comprising **high-school geometry problems , on which GPT-4o achieves only 65.3% and we successfully identified weaknesses (robustness or task generalization.) in some MLLMs.** **Through above datasets, the effectiveness and adaptability of MathCheck are validated.** Recently, we also recognize the community's eagerness in more challenging textual mathematical problems in MathCheck paradigm. In the future work, we will develop MathCheck with more difficult textual benchmark such as MATH. Thanks for your suggestion! > Data statistic. Sorry for the confusion. **Table 5 shows the variants count of each seed question.** In instance, for a seed question from gsm8k, we have 1 PU-PS vaiants and 2 PU-AJ vaiants (Answerable and Unanswerable respectively). This distribution ensures that our data is balanced and diverse. We will make the descriptions clearer to avoid confusion. > Minor modifications. Thanks for your suggestions. In our paper, MLLM means multimodal LLMs while (M)LLM means multimodal LLMs and LLMs. **Based on your suggestions, we have made several modifications, including:** 1. Combine Table 1 and Table 2 in same page for better comparison. 2. Use a bar chart in Figure 5 in order to easily see the performance delta. 3. Add x-axis label in Figure 6. 4. Remove the color fills in Figure 7. > Pass rate of MathCheck-Geo. Regrettably, we did not record the pass rates for MathCheck-Geo. However, **the same as MathCheck-GSM, we manually corrected all failed generated data in MathCheck-Geo ensuring that no error data are included in the evaluation.** Therefore, our MathCheck-GSM and MathCheck-Geo are entirely correct and the evaluation results are reliable. > Recommended: Evaluate models on MATHCHECK adapted to other reasoning tasks (i.e. commonsense reasoning and code) In Section 5, we discussed the potential application of MathCheck to other reasoning tasks and provided examples in commonsense reasoning and code generation. However, we did not scale these cases and evaluate them due to the cost. Actually, we conducted a preliminary test using GPT-4o-mini to answer commonsense reasoning variants and observed that it was indeed affected: **Original Question**：Yesterday‘s date was 4/30/2021. What is the date tomorrow in MM/DD/YYYY? Ground Truth: 5/2/2021 Predict: Tomorrow's date will be 05/02/2021. **(True)** **Irrelevant Disturbance-Problem Solving variants**: Yesterday was April 30, 2021. A week ago it was 4/23/2021. What is the date tomorrow in MM/DD/YYYY? Ground Truth: 5/2/2021 Predict: Tomorrow's date will be 05/01/2021 **(False)** **Irrelevant Disturbance-Outcome Judging variants**: Yesterday was April 30, 2021. A week ago it was 4/23/2021. What is the date tomorrow in MM/DD/YYYY? "solution": The date tomorrow will be 05/01/2021. Ground Truth: Incorrect Predict: Yes, the solution is correct. If yesterday was April 30, 2021, then tomorrow will be May 1. 2021. In MM/DD/YYYY format, that is 05/01/2021. **(False)** We observe that GPT-4o-mini successfully answers the original question but failed to respond correctly to two variants. **This highlights that reasoning consistency issues are prevalent across various reasoning tasks. MathCheck paradigm can effectively reveal such problems.**

Further response to Reviewer GGUq

> Whether the released banchmark has filtered out the questions? Yes. We manually checked all of generated data in MathCheck-GSM and MathCheck-Geo. **After that, annotators corrected each failed data. Therefore no wrong generated data are included in evaluation.** We include the detailed description of human evaluation process in the Appendix C.3, please check. Thanks for your review again. If our response can alleviate your concerns and promote your positive view of the paper, we would appreciate it if you could strengthen the recommendation.

Response to Reviewer-GGUq:

Dear Reviewer-GGUq: Thanks for your review and positive score. As our main contribution, MathCheck is a general paradigm for comprehensive mathematical reasoning evaluation and fine-grained analysis of models. We validated its effectiveness and adaptability on MathCheck-GSM(text, simple question) and MathCheck-Geo(multi-modal, high school question).In our experiments, we choose GSM8k instead of MATH because many math reasoning studies conducted on GSM8K and one specific study [1] using the private dataset GSM1K, which can further verify our analysis. As a textual math problem dataset, the process of transform MATH into MathCheck paradigm is similar to GSM8k since it primarily utilizes GPT-4o's semantic understanding and rewriting capabilities rather than problem-solving abilities. For your reference, we collected MathCheck data in MATH problems to demonstrate its transferability. Specifically, we utilized GPT-4o as rewritten model for data generation and manually checked each of them through human verification process. We find that the pass rate checked by human annotators is similar with those on GSM. Due to limited time, we collect 240 samples, and you can check these reference samples on https://anonymous.4open.science/r/MathCheck/MATH_checklist.json. Thank you once again for your review and valuable suggestions. Hopefully, this can solve your concern. [1] [A careful examination of large language model performance on grade school arithmetic](https://arxiv.org/abs/2405.00332) (Zhang et.al, NeurIPS 2024) best wishes, Authors of Paper 7051

Response to Reviewer WuK6

Thank you for reviewing our work. Below, we will address your concerns and suggestions point by point: > How to handle the low-quality generated data? We manually checked all of generated data in MathCheck-GSM and MathCheck-Geo. After that, **annotators corrected each failed data**. **This approach ensures our MathCheck-GSM and MathCheck-Geo are entirely correct and the evaluation results are reliable.** We include the detailed description of human evaluation process in the Appendix C.3, please check. > Line 431: It would be helpful to include references to the alleged works. Line 463: quote left mark is incorrect. Thank you. **We include related references to the alleged works and correct the typo(quote left mark) in the new version**, please check. > In Figure 6, could the accuracy drop be attributed to the increasing difficulty LLMs faced in generating or reconstructing task generation entries for more complex problems? All of our test data has been manually reviewed and corrected to ensure the reliability of the evaluation. Therefore, the performance drop in Figure 6 is not due to generated data correctness of difficult problems. **Under such fair comparison, the performance drop implies MathCheck better demonstrates the reasoning skills and capabilities required when problems become difficult.** > Pass rates of different complexity questions. We did not record the pass rates for different complexity levels. However, **we manually corrected all failed generated data ensuring that no error data are included in the evaluation.** Thanks for your review again! We hope our responses can address your concerns.

Response to Reviewer WuK6

Dear Reviewer, We sincerely appreciate your feedback and the time you have dedicated to reviewing our paper. During the rebuttal process, we have earnestly addressed your concerns and questions with detailed, point-by-point replies, including: - The strategy of handling incorrect generated data. - Introducing references in Section Behavior of Math Models. - Typo correction. - The reason of performance drop in Figure 6. - Clarifying the filtering process of generated data from complexity questions. Detailed updates are highlighted in blue in our revised paper. We appreciate these valuable suggestion, which enhanced the clarity and comprehensibility of our work. We hope that our response and revision adequately address your concerns. If you have any further questions or require additional clarification, please feel free to reach out. We remain fully available and happy to provide further modifications as needed. Best wishes, Authors of Paper 7051

Response to Reviewer eZPq

Thanks for your time and valuable comments. Below, we address your concerns and suggestions point by point: > About the seed question GSM8k and Geometry-QA. **The goal of MathCheck is to provide a more comprehensive of mathematical reasoning abilities at a given difficulty level, rather than simply making math problems harder to stump the model.** Base on this, we design a test checklist from reasoning robustness and task generalization. It allows users to better evaluate the math reasoning ability and conducts fine-grained analysis. Besides, it is a plug-and-play testing framework suitable for mathematical problems of different difficulty. We construct MathCheck-GSM for two motivations: **(1) validate the effectiveness of MathCheck paradigm. (2) determine whether LLMs are capable of reasoning at the grade school level while most of them achieve above 85% on GSM8k before.** Through our experiments, we observe significant performance drops in many LLMs, which validates the effectiveness of our paradigm and reveals that some LLMs rely on memorization to solve GSM8k problems. Meanwhile, we also want to evaluate models on MathCheck with multimodal math problems. To address this, we developed **MathCheck-Geo comprising high-school geometry problems, on which GPT-4o achieves only 65.3% and we successfully identified weaknesses (robustness or task generalization) in some MLLMs.** **Through above datasets, the effectiveness and adaptability of MathCheck are validated.** Recently, we also recognize the community's eagerness in more challenging textual mathematical problems in MathCheck paradigm. In the future work, we will develop MathCheck with more difficult textual benchmark such as MATH. Thanks for your suggestion! > Adding Related works. Thanks for your suggestion! We indeed drew inspiration from many related works during the design of MathCheck. We have added two new sub-sections in related work(Benchmarks of Reasoning Consistency, Strategies of Improving Mathematical Reasoning) and incorporate them into new version, please check. > About the human evaluation process. Sorry for the confusion. We selected three graduate students as human annotators, all of them are not the authors. Our human evaluation principle is the generated data should maintain the correctness of mathematical logic. For example, in the “Problem Understanding”, the generated question should not alter the logical of original question, which ensures the consistency between rewritten question and answer. After that, annotators corrected each failed data instead of discarding them. This approach ensures our dataset is entirely accurate and the evaluation results are reliable. We include the detailed description of human evaluation process in the Appendix C.3, please check. Thanks for your review again! We hope our responses can address your concerns.