OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

Summary

This paper curate a large scale (14 M qa-pairs) and high-quality math reasoning SFT datasets - OpenMathInstruct-2. At the same time, authors find some insights: Verbose solutions is detrimental to SFT performance. Data generated by strong model is better. SFT allow some low-quality solutions. Question diveristy is crucial. SFT the Llama-3.1-8B-Base on OpenMathInstruct-2 dataset achieve 67.8% performance, outperforming Llama3.1-8B-Instuct by 15.9%

Strengths

The paper introduces a large-scale math reasoning dataset, which would be highly beneficial if released. It is well-written and easy to follow. The authors offer valuable insights into data curation and supervised fine-tuning, discussing aspects like the format of chain-of-thought and the diversity of questions. Some insights are interesting and can be explored deeper.

Weaknesses

The process of dataset generation is not novel. The main advantage is the larger scale. Some findings like data generated by strong teacher is better, question diversity matters are very natural.

Questions

How do authors find or select the OpenMath CoT format? The findings that OpenMath CoT is short but performs better is interesting. I think better COT format exploration is an interesting direction. In section 3.1, aurthors discussed the LLM decontamination. Is the SFT data is processed the decontamnation? Have authors conducted experiments comparing the performance of with and without the decontamination?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Ethics concerns

NA

Summary

The paper introduces OpenMathInstruct-2, a large open-source dataset for mathematical reasoning containing 14M question-solution pairs (approximately 600K unique questions), making it nearly eight times larger than previous open-source datasets. The authors conducted systematic experiments to understand how different dataset design choices affect model performance, finding that: 1. Solution format matters - excessive verbosity can hurt performance. 2. Teacher model quality is important 3. SFT is robust to low-quality solutions 4. Question diversity is crucial. Using these insights, they created OpenMathInstruct-2 and demonstrated its effectiveness through model fine-tuning. Their OpenMath2-Llama3.1-8B model achieved 67.8% accuracy on MATH, outperforming Llama3.1-8B-Instruct by 15.9%. The authors also introduced a thorough LLM-based decontamination pipeline to ensure test set integrity. The paper concludes by releasing all code, models, and the dataset under a permissive license to support open-source development in mathematical reasoning.

Strengths

* **Ablation Studies & Methodology** - The paper excels in its systematic investigation of dataset design choices. The ablation studies on solution format, teacher model quality, and robustness to low-quality solutions are thorough and well-controlled. * **Open Science & Reproducibility** - The authors make significant contributions to open-source development by releasing their dataset, code, models, and detailed methodology. Their comprehensive documentation of prompts (in appendices D.1-D.5) and data generation pipelines enable full reproducibility. * **Math Synthetic Data Insights** - The paper provides several interesting findings that advance our understanding of LLM synthetic data generation in math reasoning. For instance, the discovery that shorter solution formats can outperform more verbose ones (OpenMath CoT vs Llama CoT) challenges conventional wisdom about detailed explanations. Similarly, their finding that SFT is robust to up to 20% incorrect solutions provides valuable guidance for dataset creation, suggesting that aggressive filtering may be unnecessary.

Weaknesses

**Insufficient Exploration of Question Generation Quality** - While the paper presents a thorough decontamination pipeline for synthetic questions, it lacks a detailed analysis of the quality and diversity distribution of generated questions. The authors don't evaluate whether their synthetic questions maintain: 1) the same quality as the original questions; 2) the same difficulty level as the original dataset 3) the same diversity (or more diverse) compared with the seed data. This might be crucial. The authors' ablation studies also show that even though the model is error-tolerant to wrong answers to some extent (e.g., 20%), the accuracy would downgrade more if there were too many error examples in the training. It would be necessary to examine the quality of the 14M generated samples. **Potential Knowledge Distillation Bottleneck** - There's a fundamental question about the approach. Though the performance is promising on a small student model (i.e., Llama-3.1-8B) with the 14M synthetic data, it is unclear whether improvement comes from primarily distilling the teacher's knowledge or improving genuine mathematical reasoning capabilities. The diminishing returns with larger student models (70B vs 8B) are consistent with distillation behavior, where the student model's capacity approaches needed to mimic the teacher's surface patterns rather than learning deeper reasoning. This limitation could be particularly important for scaling mathematical reasoning capabilities beyond the current teacher model's abilities. I'd suggest either of the experiments: - Further fine-tune the teacher model (340B) on the 14M data (which might be costly) - Generate a relatively larger-scale training set (>1M) using an 8B model and fine-tune the model on the generated data. **Incomplete Investigation of Teacher Model Selection** - The authors demonstrate that Llama3.1-405B-Instruct outperforms Llama3.1-8B-Base as a teacher model, but they don't explore the space between these two extremes. Would a 70B model be sufficient for a teacher? Is there a clear relationship between teacher model size and generated solution quality? Given the computational cost implications of using a 405B parameter model for data generation, understanding the minimum viable teacher model size would be practically valuable.

Questions

- The authors acknowledge but don't fully investigate why their OpenMath2-Llama3.1-70B model shows less dramatic improvements compared to their 8B model. They hypothesize that "our data blend or solution format might be more suited for weaker models" but don't test this hypothesis. - Have you tried other sizes of teacher models?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Summary

This paper introduces OpenMathInstruct-2, a large open-source dataset for improving mathematical reasoning in language models. The authors propose a more concise Chain-of-Thought (CoT) format, which enhances performance compared to existing methods. Through extensive experiments, they demonstrate that using a strong teacher model for data generation is more effective than on-policy data from weaker models. The study also highlights the robustness of models to low-quality data and emphasizes the importance of question diversity in fine-tuning. Overall, the paper contributes to open-source resources and offers valuable insights into optimizing mathematical reasoning in language models.

Strengths

1. Open-Source Dataset: This paper released the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license. 2. Extensive Experiments: Not only main experiments, authors did a lot ablation study about: 1. choice of data generation model, 2. robustness of SFT, 3. impact of question diversity. The results are interesting and solid.

Weaknesses

This is a solid work; I don't think there are any obvious weaknesses.

Questions

1. What do you think about a self-improving framework, such as using an 8B Llama model to generate synthetic data and fine-tune it, then repeating the process for 2-3 rounds? 2. What do you think is the reason for the inadequate performance of on-policy data?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Summary

This paper introduces OpenMathInstruct-2, an SFT recipe for improving the math reasoning of language models. With extensive engineering efforts and experiments, the authors show the importance of solution formats, the need of a strong teacher, question diversity, and the robustness to imperfect solutions. Putting everything together, the authors show that OpenMathInstruct-2 improves Llama 3.1 and performs strongly compared to other open-source models.

Strengths

I think this paper is in general a high-signal paper. Specifically: - **Well-written with high-density of details**: this paper is well-written. The flow is a naturally interleaved summary, detail, and explanations. Many important details (size of model, size of dataset, specific accuracy numbers .etc) are packed in the writing with clear explanations of the motivation. - **Well-execution**: the experiments in this paper is clear. I particularly appreciate the effort and discussion about decontamination, which essentially strengthens this paper quite a bit. - **Sufficient signal from experiments**: the model developed in this paper is not top 1 in the leaderboard (but understandable because Qwen uses more resources). But the signal sent by the experiments (tolerate to imperfect annotation, importance of question diversity .etc) is clear.

Weaknesses

There are two conclusions draw by the authors that I am not so sure: - **Format of the data**. In general I agree that excessively verbose CoT may not be good for either performance or user preference, but it is not very clear how exactly the CoT proposed in this work contribute to the improved accuracy of reasoning. The higher numbers reported by the author is a good observation but not the mechanism behind. In addition, existing successful models like O1, in many cases, also offer verbose CoT, yet its performance is remarkable. In general, I tend to believe the level of detail that should be included in the CoT may still need further investigation, and I do see mixed signals favor or not favor verbose CoT. - **Limited domain**: the authors still use GSM8K and MATH as the seed data — they do generate new questions but the seed is still GSM8K and MATH. The author test the performance on more challenging tasks (AMC and AIME), which is a nice practice. But I do feel that the field is moving forward to wider domains (not just math but also other STEM) and higher level (not just high-school but college). For this work I think it is sufficient to just use GSM8K and MATH, but I would suggest the authors to move forward in their future endeavors — only focusing on these two datasets would limit the authors from making higher impact.

Questions

In Table 4, the authors compare with Qwen 2.5, Mathtral, NuminaMath and DS-Coder. I understand these are strong open-source baselines so comparing with them is sufficient to see the signal, but I would still encourage the authors include more OSS/ proprietary models for further comparison (even other models are much weaker or stronger).

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4