MAmmoTH2: Scaling Instructions from the Web

Summary

The paper proposes a 3-stage pipeline to harvest ex-large-scale instruction data from the pre-training web corpus to enhance LLM reasoning, which involves 1) recalling relevant documents, 2) extracting instruction-response pairs using LLM, and 3) refining the extracted pairs by completing the intermediate reasoning steps using LLM. The paper - proposes an effective pipeline to synthesize large-scale high-quality instruction data, especially for reasonable prompts and reliable answers; - empirically validates the effectiveness of scaling up instruction data for reasoning tasks; - builds `MAmmoTH2-Plus` models, achieving performance superior to or comparable with previous SotA on various reasoning datasets; - provides a ex-large-scale instruction dataset for reasoning tasks, *WebInstruct*, as unique public data resource; - conducts extensive ablation studies, providing many insights like: - SFT loss is better for LM loss (at least when evaluated on QA tasks); - refining extracted instruction pairs by completing the intermediate reasoning steps is significantly helpful; - using multiple LLMs to refine the instruction data is usually better than a single LLM; - “Education” data (exam-style) are usually better than “Forum” data (discussion-style) (at least when evaluated on QA tasks); - even benchmarks conventionally thought very relevant might conflict with each other (GSM & MATH in Table 5), implying limited generalization of LLMs.

Strengths

- The scaling effect of instruction data is an important empirical question. The paper is the first to scale instruction data to 10M pairs, showing the feasibility and effectiveness of scaling up instruction data (for reasoning tasks). - Synthesis of high-quality prompts and answers is important for further data augmentation but rather under-explored. The paper finds an effective method to synthesize reasonable prompt and relatively reliable answers by harvesting from web corpora. - `MAmmoTH2-Plus` models achieve performance superior to or comparable with previous SotA on various reasoning datasets. - Extensive experiments are conducted on various base models and especially diverse challenging reasoning benchmarks, instead of easy ones with limited scope (e.g. many benchmarks similar to GSM8K), convincingly validating the method's effectiveness. - Many insightful and useful observations in ablation studies (as mentioned in the summary). - The paper is generally well written to be clear and detailed.

Weaknesses

- It might need further consideration about **whether training on *WebInstruct* is compatible with or necessary to be added to existing training pipelines to achieve the best final performance (for reasoning tasks)**. The paper achieves its best performances (`MAmmoTH2-Plus`) with a 2-stage instruction tuning on pre-trained models but doesn’t involve continual pre-training, which should be rather important for models’ reasoning abilities as proved by works like DeepSeek-Math. Pre-training and RL should be out of this work’s scope. But it would be better to further clarify the impacts of 1) continual pre-training, 2) training on *WebInstruct*, 3) final fine-tuning on additional instruction datasets and their combinations. - Table 7 shows the performance on reasoning benchmarks of applying 2/3/2+3 on Mistral-7B/Mixtral-8x7B. But **the comparison might be a little unfair**: the domains of the “Public Datasets” are wider than those of the *WebInstruct* with the code generation dataset *Code-Feedback*, but the benchmarks only involve mathematical and scientific reasoning in natural language, which might underestimate the performance of “Public Datasets”, considering the possible confliction between code generation and reasoning in natural language. It might be better to remove *Code-Feedback* from the “Public Datasets” to compare with *WebInstruct*. - **To consider 1) continual pre-training**, it is impossible to conduct by yourselves, but a possible workaround could be to make full use of the resources provided by DeepSeek-Math: DeepSeekMath-7B is continually pre-trained from DeepSeek-Coder-Base-v1.5. By comparing performances on reasoning benchmarks of applying 2/3/2+3 on DeepSeek-Coder-Base-v1.5/DeepSeekMath-7B and the two models themselves, a more comprehensive study on the impacts of these training stages can be done. - Table 7 shows that, for strong Mixtral-8x7B, the gains of adding *WebInstruct* to “Public Datasets” is marginal, implying that **the effect of *WebInstruct* for strong base models might be limited**. --- # After rebuttal and discussion The authors resolved most concerns and validated that MAmmoTH2 can efficiently substitute continual pre-training in the standard SotA pipeline. The limitation is that MAmmoTH2 fails to combine with continual pre-training to effectively push forward the upper limit. I decide to change my score to 8.

Questions

Suggestions: - The refinement step is important and the current setting can be seen as distillation from strong models (Mixtral-22B×8 and Qwen-72B). The method could be more promising if it could help self-improvement/weak-to-strong generalization. I highly recommend adding experiments of training Mixtral-22B×8 and Qwen-72B or stronger models in future versions. Confusions: - Are training data sizes in experiments for Table 5 controlled to be comparable? - What does the Data Source “Base“ mean in Table 5?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitations of this work are acceptable and the authors point out potential directions to address the limitations for future works.

Summary

The paper introduces MAmmoTH2, a novel approach to instruction tuning for large language models (LLMs) by harvesting naturally existing instruction data from the web. The authors develop a three-step pipeline (recall, extract, refine) to collect 10 million high-quality instruction-response pairs without relying on costly human annotation or GPT-4 distillation. Fine-tuning LLMs with this dataset significantly improves performance on reasoning benchmarks. The MAmmoTH2-Plus model, further tuned on public instruction datasets, achieves state-of-the-art results on multiple benchmarks.

Strengths

- Demonstrates a cost-effective way to collect large-scale, high-quality instruction data from the web. - Significant performance gains on reasoning benchmarks, with MAmmoTH2 models outperforming existing models. - Comprehensive evaluation across multiple benchmarks, showing robust improvements.

Weaknesses

- The approach primarily combines existing methods (data recall, extraction, refinement) rather than introducing fundamentally new concepts or techniques. - More explicit comparison with prior work is needed to highlight the unique contributions and differences of this approach. - The quality and diversity of the collected data heavily depend on the web sources, which may introduce biases or inconsistencies.

Questions

- How does MAmmoTH2 compare directly with other methods that use synthetic or human-annotated data in terms of data quality and model performance? - What measures were taken to ensure the quality and relevance of the extracted Q-A pairs from the web? - How does the model address potential biases in the web-sourced data, and what steps were taken to mitigate these biases?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors address some limitations of their approach, such as the dependency on web data quality and the challenges in maintaining the diversity and relevance of the instruction data. However, a more detailed discussion on potential biases introduced by web data and the ethical implications of using such data could strengthen the paper.

Summary

This paper proposes an approach to automatically harvest large-scale instruction data from pre-training corpora for reasoning tasks. The main steps include: (1) Recall: training a fastText model to recall relevant documents from the pre-training corpus, similar to DeepSeekMath; (2) Extract: using open-source models with few-shot prompting to extract question-answer pairs from the recalled documents; (3) Refine: prompting open-source models to remove noise, adjust formats, and complete the reasoning process for the extracted question-answer pairs. Using this method, the authors harvested 10 million instruction data and trained MAmmoTH2 models. Without relying on closed-source models, MAmmoTH2 achieves excellent performance on various reasoning tasks.

Strengths

- Method: The motivation is clear, and the idea of automatically extracting instruction data from web data is novel, simple, and scalable. - Experiments: The experiments and evaluations are comprehensive and achieve good results. - Well Written: The paper is very easy to understand. - Reproducibility: The authors have open-sourced part of the corpus, models, and evaluation scripts to ensure the reproducibility of the results.

Weaknesses

1. Effectiveness: I wonder if the WebInstruct approach can further improve the performance of state-of-the-art domain models. For example, DeepSeekMath achieved good results by only training on recalled documents and fine-tuning on high-quality data (MATH: DeepSeekMath-7B-Instruct 46.8% vs. MAmmoTH2-7B-Plus's 45.0%). Moreover, since the models have already been trained on SFT data, comparing only the few-shot performance is not comprehensive enough. I suggest also comparing the performance of the Plus version trained with high-quality "additional instruction datasets" for most of the experiments. Consider supplementing the following results: - Recall + Plus: Directly train on the 18M recalled documents and fine-tune a Plus version to verify if the "extract + refine" steps have significant benefits. - Recall + Extract + Plus: Directly train on the extracted QA (Fig.5, Extracted QA) with LM/SFT loss and fine-tune a Plus version to verify the benefits of the refine step. - In Fig.5, I also recommend reporting the performance after fine-tuning the Plus version for SFT loss vs. LM Loss. 2. Lack of method details: - For example, the code for the recall stage and the prompts used for extraction and refinement could be included in the repository or appendix. - In Sec. 5.1, I suggest explicitly defining the SFT Loss to help more readers understand it clearly. By "SFT Loss", I understand the authors mean "masking the loss of instruction input", right? 3. Scalability: - The effectiveness of WebInstruct constructed using small models is unknown for larger models; moreover, this approach is difficult to apply to models with hundreds of billions of parameters due to high inference costs. - During refinement, the model generate missing explanations. Have you observed and quantified the hallucination phenomenon? If present, such incorrect reasoning processes can negatively impact model training, such as increasing hallucination/bias, especially if the corpus is used for larger models. 4. Minor points: - Some citations are missing for baselines in Table 2, e.g., Gemma, Abel, and Rho-1. - How can the WebInstruct approach be extended to more general domains? What other issues need to be addressed? - A concurrent work, Jiuzhang3.0 [1], is quite similar in motivation and method. It would be better to discuss and compare with it. What are the advantages and issues of MAmmoTH2 compared to Jiuzhang3.0? --- [1] Zhou, Kun, et al. "JiuZhang3. 0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models." arXiv preprint arXiv:2405.14365 (2024).

Questions

See weaknesses.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have addressed most of the limitations. The limitations section can be further improved by referring to the weaknesses part.

Summary

This paper proposes a method to synthesize instruction tuning data at scale from the pretraining web corpus. The proposed method first recalls relevant documents from the corpus, and then extracts QA pairs, and finally refines the extracted QA pairs with an LLM. The synthesized instruction data proves to be helpful in enhancing the model’s reasoning abilities compared with instruction tuning data from other sources.

Strengths

1. The proposed method is novel and effective. 2. The authors conduct extensive experiments to demonstrate that it’s possible to synthesize tuning data from unsupervised text corpus to build strong LLMs that outperform models trained with data collected in existing paradigms. 3. The paper is well-written and easy to follow. The code and data are released, which will serve as high-quality resources for research and building strong LLMs.

Weaknesses

There lacks a discussion and comparison with a related work “Self-alignment with Instruction Backtranslation” (Li et al., ICLR'24) which also synthesizes instruction tuning data from unlabeled corpus.

Questions

LLMs are used in the “extract” and “refine” steps in the proposed pipeline for generating and editing instruction tuning data. Will the choice of LLMs introduce bias into the synthesized data (especially compared with distillation-based methods)?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations in Appendix H and societal impacts in Appendix I.

Is the Extract and Refine Process Necessary?

Thank you to the author for supplementing the results and replying. However, my main concern has not been addressed, namely **whether the (b) extract and (c) refine processes introduced by MAmmoTH2 provide significant benefit or necessity to the existing data pipeline**. As mentioned in my previous comments, DeepSeekMath achieved higher results using only **(a) recall and (d) sft** (MATH: DeepSeekMath-7B-Instruct 46.8% vs. MAmmoTH2-7B-Plus's 45.0%). Therefore, I once again suggest that the author design controlled experiments and provide the following results (for models >=7B) to demonstrate whether (b) and (c) are necessary: 1. **(a) Recall + (d) SFT**: Directly train on the 18M recalled documents and fine-tune a Plus version to verify if the "(b) extract + (c) refine" steps have significant benefits. 2. **(a) Recall + (b) Extract + (d) SFT**: Directly train on the extracted QA (Fig.5, Extracted QA) with LM/SFT loss and fine-tune a Plus version to verify the benefits of the (c) refine step. 3. In Fig.5, I also recommend reporting the performance after **fine-tuning the Plus version** for SFT loss vs. LM Loss.

Follow up on Ablation Results

We appreciate **Reviewer Jcxm and k2fV**'s insightful comments and have conducted **all the required ablation studies** before the end of the discussion period (bingo!). Our experiments, summarized in the table below, demonstrate the effectiveness and efficiency of our proposed pipeline. ### Experimental Setup - Base: DeepSeek Coder V 1.5 7B - (a) MAmmoTH2's recall documents: 18M documents, 28B (14B tokens * 2 epochs) - (a') DeepSeek Math's CT corpus: 500B tokens (120B math + others for multiple epochs) - (b) Extracted instruction-response pairs from (a): 7B (3.4B tokens * 2 epochs) - (c) Refined instruction-response pairs from (b): 10B (5B tokens * 2 epochs) - (d) MAmmoTH2 Additional public SFT: 2B (1B tokens * 2 epochs) - (d') DeepSeek Math SFT: ~1B tokens | | Setting | Model | #Train Tokens | TheorQA | MATH | GSM8K | MMLU-S | BBH | ARC-C | AVG | |---|---|---|---|---|---|---|---|---|---|---| | | All evaluations are held-out | | | | | | | | | | | 1 | Base | DeepSeek Coder v1.5 7B | - | 18.3 | 22.3 | 47.9 | 47.0 | 53.5 | 62.4 | 41.9 | | 2 | Base + (a) | - | 28B | 23.5 | 30.3 | 60.3 | 53.3 | 55.5 | 69.4 | 48.7 | | 3 | Base + (a) + (b) + (c) | MammoTH2-DS | 10B | **27.8** | 33.8 | 64.0 | **56.9** | 58.5 | **72.8** | **52.3** | | 4 | Base + (a’) | Deepseek Math Base | 500B | 25.3 | **34.0** | **64.2** | 56.4 | **59.5** | 67.8 | 51.2 | | | | | | | | | | | | | | | All held-out except GSM and MATH | | | | | | | | | | | 5 | Base + (d) | - | 2B | 23.5 | 37.2 | 77.5 | 52.0 | 59.8 | 66.9 | 52.8 | | 6 | Base + (a) + (d) | - | 30B | 27.2 | 39.2 | 79.2 | 55.6 | 60.3 | 71.5 | 55.5 | | 7 | Base + (a) + (b) + (d) | - | 9B | 27.3 | 38.6 | 78.5 | 54.2 | 60.5 | 70.4 | 54.9 | | 8 | Base + (a) + (b) + (c) + (d) | MAmmoTH2-DS-Plus | 12B | **30.1** | 43.8 | 80.1 | **59.5** | **61.0** | **73.2** | **58.0** | | 9 | Base + (a’) + (d’) | DeepSeek Math Instruct | 501B | 23.7 | **44.3** | **82.9** | 59.3 | 55.4 | 70.1 | 56.0 | | 10 | Base + (a’) + (a) + (b) + (c) + (d) | MAmmoTH2-DS-Math-Plus | 512B | **31.5** | **45.2** | 80.2 | **60.5** | 62.0 | **76.4** | **59.3** | | 11 | Base + (a’) + (d) | - | 502B | 28.8 | 43.4 | 80.6 | 60.1 | **62.2** | 72.5 | 57.9 | ### Key Findings 1. **Cost-Effectiveness**: Our pipeline achieves superior overall performance compared to DeepSeek Math models, both before **(Row 3 vs Row 4)** and after additional SFT **(Row 8 vs Row 9)**, while using **significantly fewer tokens**. While DeepSeek Math models show slightly higher results on math-specific benchmarks, our approach demonstrates better performance on a broader range of reasoning tasks, including STEM-related benchmarks. 2. **“Extraction Step” Efficiency**: The "Extract" step in our pipeline leads to a more cost-effective approach, using fewer tokens while maintaining comparable performance **(Row 6 vs Row 7)**. 3. **Refinement Importance**: The "Refine" step proves to be crucial, significantly improving answer quality by adding missing explanations and chains of thought, resulting in substantially better performance **(Row 7 vs Row 8)**. We believe these comprehensive experiments address the reviewer's concerns and further underscore the merits of our approach, especially the “extract” and “refine” steps. We really appreciate the reviewer's feedback, which has led to these valuable insights and a stronger demonstration of our pipeline's effectiveness. Feel free to let us know if you have further comments!