Summary
This paper presents a new dataset called MathOdyssey, which aims to evaluate the reasoning abilities of large language models (LLMs). The dataset consists of 387 problems, including 148 at the Olympic level, 101 at the university level, and 138 at the high school level. The problems cover several subjects with three answer formats: true/false, multiple choice, and open answers.
The authors evaluate LLMs' math reasoning performance on MathOdyssey using GPT-4 as the answer judger in a zero-shot manner, providing it with specific instructions. They conduct their experiments with seven closed-source LLMs and one open-source model, Llama-3-70B. Their findings reveal that the Llama-3-70B model still falls short when tackling more complex problems.
Strengths
The proposed MathOdyssey dataset is novel and may be somewhat useful to certain researchers. While it introduces a variety of challenging problems, the experimental results provide a rough indication of the performance of closed-source LLMs. Overall, the dataset and findings suggest potential (but limited) usefulness to offer insights into LLM capabilities in mathematical reasoning, albeit with room for more comprehensive analysis.
Weaknesses
MathOdyssey **offers no clear advantages over existing benchmarks**, which may limit the usefulness and contribution of this paper.
- Compared to existing datasets, MathOdyssey is **limited in size**, containing only 387 problems, whereas datasets like GSM8K and MATH include 1,319 and 5,000 problems, respectively. This limitation might impact the reliability of accuracy in ranking the mathematical reasoning abilities of different LLMs.
- **The "difficulty levels" within MathOdyssey are not well-defined**. Although it claims to cover comprehensive levels of math problems, it includes only three educational stages. In contrast, the MathBench [1] dataset offers a wide range of problems, spanning from primary school to university level. By the way, some datasets define the difficulty level as a rating (e.g. an integer number)
- While the authors claim to have diversified answer types, MathOdyssey **only encompasses three distinct answer types**. OlympiadBench [2], however, incorporates a more fine-grained variety of answer types.
- Although MathOdyssey includes several subjects, **the number of testing examples within each subject is relatively small, with many subjects containing fewer than 10 examples**. This limitation may lead to inaccurate analyses across different subjects.
**The experiments are not comprehensive and compelling**:
- The evaluation process is flawed because the authors use GPT-4 as the judge for answers in a zero-shot manner. However, **it is unclear how often this judgment aligns with human evaluators**. An analysis of judgment errors is necessary, and I recommend considering rule-based matching.
- They **include only one open-source LLM, Llama-3-70B** in experiments, which is not comprehensive. The authors should include more open-source LLMs, including both general-purpose chat models and math-specialized LLMs.
- The results and analysis are not compelling, as the reliability of GPT-4's judgment is uncertain. Additionally, **conducting an error analysis could provide valuable insights into how LLMs get wrong in solving math problems**.
This paper **is poorly written** and either **lacks important details or makes inaccurate claims or claims without proper citations** (see Questions).
[1] Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., ... & Chen, K. (2024). MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark. arXiv preprint arXiv:2405.12209.
[2] He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., ... & Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
Questions
Questions and Concerns:
- Line 103-105: The authors claim, "The key distinction of our dataset is its expert-driven creation, which minimizes the risk of data contamination." In lines 106-107, they state, "The dataset has not been used for training by LLMs." To substantiate these claims, the authors should conduct a contamination detection analysis, which has not been done.
- The term "open-answers" is mentioned without a clear definition. Can these open-answers be further classified into more fine-grained types, similar to the classification in OlympiadBench [1]?
- Given that GPT-4 is used as the answer judge for all experiments and analyses, it is necessary to conduct human evaluations to determine the reliability of GPT-4 as a judge.
- The evaluation prompt requires output in JSON format. However, some LLMs are fine-tuned to produce answers delimited by "boxed". Have you investigated whether LLMs conform to the expected format?
- Have you considered incorporating difficulty levels like MATH? Some high school problems might be easier than certain university-level problems.
- Results on More Open-Source LLMs: Can you provide results from more open-source LLMs?
Other Issues:
Lack of Appropriate Citations:
- Lines 39-41: "has achieved more than a 90% success rate"
- Lines 44-45: "technological advancement but a crucial step toward developing more general and capable artificial intelligence systems"
- Lines 51-52: "Moreover, a significant obstacle is that many existing datasets might have been included in the training phases of these models, potentially skewing performance metrics."
- Lines 320-322: Citations or links for LLMs are missing.
Inaccurate Statements:
- Lines 47-49: "it remains uncertain how well they handle more complex mathematical challenges, such as those found in university-level courses and competitive high school mathematics." In fact, datasets like MATH and MathBench [2] include competitive high school mathematics and university-level courses, respectively.
- Lines 51-52: "Moreover, a significant obstacle is that many existing datasets might have been included in the training phases of these models, potentially skewing performance metrics." There are, however, many variants of existing datasets that address this issue, such as GSM-1K [3], which also weakens the contribution of this paper because MathOdyssey is limited in size.
- Table 2: The number of examples for GSM8K should be 1319.
[1] He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., ... & Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
[2] Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., ... & Chen, K. (2024). MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark. arXiv preprint arXiv:2405.12209.
[3] Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., ... & Yue, S. (2024). A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332.