Response to Reviewer ZpsQ
Dear Reviewer ZpsQ,
We hope this comment finds you well! Thank you for your comments! We will address your concerns one by one as follows:
> How do you ensure that changes to specific values within a problem do not fundamentally alter its nature or required solution approach? Given that small modifications can impact mathematical properties such as convergence and factorization, are there any safeguards or verification steps to confirm that randomized problem variants retain similar problem structures, difficulty levels, and solution approaches as the original?
Thank you for your question! In fact, the problems in our grading system are generated by programs, with specific range for variables to guarantee the required solution approach and correctness (As implicitly mentioned in Appendix B.1 Line 1024 in the original manuscript). **We have made it more clear and added two examples in Appendix A.1 and B.1 (Figure 7 and 8, Listing 1) to show this point**. For your convenience, we briefly discuss these examples here:
- For question $\int x^5 cos(x^3) dx$, when we generate this version of problem, the following ranges of variables are considered: \$a = random(2, 10); \$ expnt = -1 + 2* \$ a;. This meticulous setup maintains the consistency of the concepts, techniques, and solutions
involved in different versions of each problem.
- For question to determine the convergence of series $\sum_1^\infty \frac{sin (n)}{n^{6.5}}$, when we generate this version of problem, the following range of variable is considered: $a = random(2,10,0.1). This setup makes sure this series is convergent.
> Precision in Numerical Calculations: Classifying the example in Table 12 as a "numerical calculation error" might be overly strict, given that the minor difference between 12305.72 and 12305.670999 likely falls within acceptable rounding tolerance. Unless the benchmark is meant to test extreme precision, such small discrepancies shouldn’t necessarily indicate an error. Accuracy assessments could allow for minor rounding differences or specify a standard precision level (e.g., significant figures or decimal places) for intermediate calculations to ensure fair evaluation.
Thank you for pointing this out! We apologize for the wrong example provided. The example in Table 12 also surprises us when we see your Comments: we set the relative precision error to be 1e-3 in our evaluation. After checking, we find that one of the authors picked a wrong example whose answer was judged to be correct. We were rushing to the deadline and did not discover this error. Now we have updated the example in Table 30 in our revised manuscript.
> Would it be feasible to incorporate a tolerance level for minor discrepancies in numerical answers, especially those influenced by floating-point precision errors? For instance, a defined rounding tolerance (e.g., within 0.01%) could help accommodate acceptable computational variations and more accurately reflect the model’s reasoning accuracy.
Yes. We **have set the relative tolerance level to 1e-3 in our evaluation**, which is quite looser than the criterion for OlympaidBench [1] (1e-5) and OlympicArena [2] (1e-8). We have also added a figure (Figure 9) to illustrate the relative numerical error for OpenAI-o1-mini in Appendix E. We can see from the figure that 1e-3 is a suitable choice.
We acknowledge the potential issue of numerical calculation errors in LLMs. We believe that integrating tool-supported reasoning could be a promising approach to mitigate this issue in the future [3, 4].
We hope our changes have addressed your concerns. Feel free to further discuss with us at this stage!
Best Regards,
Authors
[1] He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., ... & Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
[2] Huang, Z., Wang, Z., Xia, S., Li, X., Zou, H., Xu, R., ... & Liu, P. (2024). OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI. arXiv preprint arXiv:2406.12753.
[3] Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Huang, M., ... & Chen, W. (2023). Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452.
[4] Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., ... & Zhang, Z. (2024). Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.