DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving

Summary

This paper synthesizes a math reasoning dataset with a designed way of rejection sampling. Many base models show performance improvements on math reasoning tasks after instruction-tuning on this dataset. They promise to release the dataset and models.

Strengths

Their curated dataset achieves relatively good instructing-tuning performance with least data amount compared to other baselines. The dataset will be released.

Weaknesses

1. The proposed sampling technique is trivial and incremental, when comparing with previous works, e.g., the uniform method is used in ToRA, and the prop2diff method is used in MARIO. 2. There’s little improvement or even performance drop when tuning Mistral-7B and DeepSeekMath-7B compared to other baselines. As mentioned in the analysis section, this dataset is somehow replaceable by math-specific continual pre-training + supervised fine-tuning (SFT). 3. The major concern is that even the paper claims the proposed dataset is smaller, however, the LLM used to synthesize the smaller dataset is `DeepSeekMath-7B-RL`, which is trained on a larger SFT dataset. An alternative and reasonable response generation method should be leveraging the `DeepSeekMath-7B-Base` with proper prompting, as `DeepSeekMath-7B-Base` has not been supervised fine-tuned.

Questions

1. What’s the query coverage ratio on MATH training set constructed by Prop2Diff? 2. Any figure or statistics to show the difficulty distribution of your DART dataset? 3. Any case studies to show the generated responses of your DART dataset? How do you extract answers in the raw response? Responded texts from the LLM are quite likely not to follow your instruction as you apply such a high temperature in sampling process. It’s not likely that simple pipelines, such as regular expressions can achieve this.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

1

Limitations

NA

Summary

The paper introduces Difficulty-Aware Rejection Tuning (DART), a novel approach for enhancing the mathematical problem-solving capabilities of large language models (LLMs). Traditional methods often produce datasets biased towards easier queries, limiting the models' ability to learn from challenging examples. DART addresses this by allocating more sampling trials to difficult queries during the data synthesis phase. The authors created two strategies, Uniform and Prop2Diff, to ensure a balanced representation of easy and difficult queries. Using only open-weight models, the authors generated new, smaller datasets that prioritize difficult queries.

Strengths

1. The DART method effectively addresses the bias towards easy queries in traditional rejection sampling, which is a significant contribution to the field. 2. The paper provides a thorough analysis of the biases in existing datasets and clearly explains how DART mitigates these issues. 3. The authors plan to make their datasets and models publicly available, contributing valuable resources to the research community.

Weaknesses

1. The success of DART relies heavily on the ability of models to generate correct responses for difficult queries, which may not always be feasible for extremely challenging problems. 2. While the focus on difficult queries is commendable, the quality of the generated responses for these queries needs to be high to truly benefit the training process. The paper does not provide a detailed analysis of the quality of these responses. 3. The approach's reliance on extensive sampling for difficult queries might pose scalability issues, particularly for very large datasets or models with limited computational resources.

Questions

1. How to be aware of difficulty if not labelled 2. How to choose $k_u$。 3. The details of Prop2Diff is missing. How many samples were generated for each difficulty level? What is the equation for generating numbers?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitation is mentioned in the conclusion section.

Summary

The paper proposes a rejection sampling pipeline for automatically generating SFT data, emphasizing that harder data requires more trials. The difficulty is heuristically determined using the ratio of incorrect trials for each question. Experiments demonstrate that this method can outperform traditional rejection methods on various math benchmarks.

Strengths

- The experiments are solid, showing significant improvements over traditional rejection methods. - The paper is clearly written and easy to follow.

Weaknesses

The proposed Prop2Diff strategy lacks innovation. Assigning more budget to more complex questions in data synthesis is a common practice. For instance, in [1], which successfully annotated 83.1% of MATH questions, it is evident that harder problems were allocated more budget in rejection sampling. [1] also indicates that fewer and harder data can significantly and efficiently improve performance. The authors should discuss the differences between their approach and the one used in [1] more thoroughly. [1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

Questions

Could you elaborate on how your approach differs from the rejection sampling strategy used in [1]? [1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper presents an approach to improving the performance of LLMs in mathematical problem-solving. The authors identify that current datasets synthesized using proprietary models like GPT-4, are biased towards easier queries. To address this, they introduce Difficulty-Aware Rejection Tuning (DART), which allocates more trials to difficult queries during data synthesis. This method generates datasets focusing on difficult queries using an open-weight model, DeepSeekMath-7B-RL, without relying on proprietary models. The authors demonstrate that models fine-tuned on DART-Math datasets significantly those fine-tuned on traditional datasets across various mathematical benchmarks, and beat the best baseline by average of roughly 3-4%

Strengths

- Technically solid paper with state-of-the-art results. - Mostly well-presented and easy to understand. - Comprehensive experiments and analysis. - Decent impact in improving mathematical capabilities of LLMs, with the authors publicly releasing their dataset. - By using an open-weight model, DeepSeekMath-7B-RL, the authors eliminate dependency on proprietary models like GPT-4, making the approach more accessible.

Weaknesses

1. It is unclear how the hyperparameters of the baseline, VRT (vanilla rejection tuning), were tuned. For instance, as mentioned in Appendix A.2, sampling temperature is searched from 0.3 to 1.7 for DART. Was the same procedure used for VRT? Another caveat is the need for extensive hyperparameter tuning compared to baselines. Were similar extensive procedures for tuning performed for other baselines? 2. It is unclear if the improved performance of the proposed method is due to difficulty or the topic of the problem. For instance, LEVEL 5 Math problems may have a higher number of geometry questions (or at least their fail rate is higher, resulting in fewer samples in VRT). An analysis of topic-wise performance comparing DART and baseline methods may clarify this. **Minor Weaknesses: ** 1. It is unclear how much advantage the method would provide in the case of other multi-iteration fine-tuning methods such as STaR and v-STaR. For instance, it is possible that after multiple iterations, VRT performs similarly to DART, since a higher number of samples will be collected from even the hard problems in second or further iterations. 2. The data synthesis is only done using the DeepSeekMATH-7B model. It is unclear why this model was chosen. Previous methods using VRT-like methods typically use the same model for synthesis and generation. Thus, higher results in smaller models such as Llama-8B may partly be due to the use of stronger models' reasoning chains, making it similar to a distillation method.

Questions

1. The authors use "fail rate" as a metric to measure difficulty. However, has any analysis been performed to measure how good an estimate it is of actual model accuracy? 2. In line 138, "we sample till correct responses for each query is proportional to its difficulty score," does it mean linearly proportional? 3. To the best of my knowledge, previous works usually use lower temperatures in the range of 0.7-1. However, the authors found 1.6 to be effective. Do the authors have a comparison of results between using a more standard temperature (e.g., 0.7 or 1) compared to 1.6? 4. In Line 253, the authors state: "We hypothesize that this is due to these models’ extensive pretraining on mathematical content." Do the authors have more points to substantiate this? For instance, it could be partly due to a slightly weaker or similar model being used to generate synthetic data. Further, the hypothesis: "This pretraining likely covers most skills that could be learned from the GSM8K and MATH training queries" may not be correct, since, at least for Llama2-70B, the model capacity should not be a bottleneck to achieving higher scores on MATH (e.g., Qwen models). Can the authors provide a more detailed reasoning behind this hypothesis?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Major Limitations are addressed in paper.

Kind Reminder before the End of Discussion

Dear Reviewers, We sincerely appreciate your feedback. Now, we would like to send you a friendly reminder that the discussion stage will be complete soon. If you have any additional concerns or questions, please do not hesitate to let us know. We are more than willing to discuss any further suggestions or issues you might have. Best regards, The Authors