AlphaMath Almost Zero: Process Supervision without Process

Summary

The paper proposes an innovative framework named AlphaMath, aimed at enhancing the mathematical reasoning capabilities of large language models (LLMs) without relying on expensive human annotations from domain experts or GPT-4. The authors leverage Monte Carlo Tree Search (MCTS) to allow a pre-trained LLM to autonomously generate process supervision and step-level evaluation signals, integrating a value model with the LLM. At the inference time, the authors propose a step-level beam search strategy, which assists the policy model in navigating more effective reasoning paths. Experimental results demonstrate that AlphaMath achieves comparable or superior results to previous state-of-the-art methods on various datasets without process supervision. The application of MCTS in LLMs is now somewhat popular, with several highly relevant papers, such as [1] https://arxiv.org/pdf/2406.06592, [2] https://arxiv.org/pdf/2405.00451 and [3] https://arxiv.org/pdf/2309.17179] (note that [1, 2] are published after neurips deadline, so no need to perform comparison, It's just added for reference.)

Strengths

1.The proposed method effectively eliminates the necessity for human or GPT-4 annotation, resulting in significant efficiencies in both financial and temporal aspects. 2. The utilization of Monte Carlo Tree Search (MCTS) can enhance the performance of LLMs in reasoning tasks by facilitating the search for optimal paths. Additionally, training with trajectories derived from MCTS is a commendable approach. 3. The introduction of step-level beam search presents a promising strategy to reduce computational costs.

Weaknesses

1. The paper's results are all program-aided, a fact not clearly identified in the text. Consequently, the comparison with non-program-aided LLMs may not be fair enough. 2. The benchmark only includes GPT models in the 'Proprietary Models' section. It would be beneficial to include other LLMs such as Gemini and Claude for a more robust evaluation. 3. An image illustrating the Step-Level Beam Search (SBS) would greatly aid in explaining the algorithm.

Questions

Check other sections.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

1. The framework does not generate a reward model that can facilitate other LLMs, which potentially limits its applicability. 2. The framework cannot be extended to tasks with open-ended outputs, limiting its potential use cases.

Summary

The authors propose the AlphaMath method, which leverages Monte Carlo Tree Search (MCTS) to improve the mathematical reasoning capabilities of large language models (LLMs) without requiring expensive human or GPT-4 annotations for process supervision. The framework integrates a value model with the LLM to generate and evaluate reasoning steps autonomously. Experimental results demonstrate that AlphaMath achieves comparable or superior performance to baseline methods on both in-domain and out-of-domain datasets. To AC and the authors: The detailed comments are mainly covered in the **Questions** section. Please refer to the section for details. Generally speaking, I would consider the paper as very close to the threshold. If the authors can properly address the concerns in the below section, this could be a valuable study in this area.

Strengths

MCTS is a powerful algorithm for complex planning. Mathematical reasoning is a very important but challenging area. Even the most advanced LLMs nowadays can easily make mistakes when solving a mathematical problem. There are many recent attempts using MCTS on mathematical tasks, and this paper is a good one among many. The empirical results demonstrate the effectiveness of the proposed method.

Weaknesses

Many presentations in the paper are somewhat unclear, requiring further clarification and elaboration. There are also some minor over-claims in the introduction. There are many modifications comparing to the original AlphaGo and Zero paper but the authors typically do not provide explanations for such changes. Some tables and figures are confusing. Please see the **Questions** section for the details. One thing I'd like to especially point out is, the title "almost zero" is kinda misleading. One would interpret it as using very little data, e.g., a few seed data. However, the method would still need the full training set of GSM8K/MATH.

Questions

1. The authors use Table 1 and corresponding statements in the main body (line 25) to claim that *the annotation of mathematical solutions primarily relies on domain experts or GPT-4 in current efforts*. This claim is inaccurate, as multiple studies employ methods such as the Monte Carlo method [1,2] or even Monte Carlo Tree Search (MCTS) [3,4,5] to automatically annotate mathematical solutions. It is totally understandable that some related studies might have been missed, especially those posted after NeurIPS submission [3,5]. However, there are studies [1,4] published last year that should have been properly considered. I recommend that the authors revisit their claims and appropriately cite and discuss works that do *annotate mathematical solutions without human intervention or GPT-4*. This paper already presents numerous promising contributions, which I am very impressed; therefore, it is advisable to carefully avoid over-claiming in minor aspects that may negatively affect the first impression of readers and researchers familiar with this field. 2. In line 76, the authors write *the solution process can be broken into multiple reasoning steps (e.g., segmenting the solution based on distinct stages or simply on a period).* What exact step segmentation strategy do the authors use in this study? 3. In line 118, the authors write, *we employ sampling generation with higher temperature to ensure diversity* in generating actions for a step. This method requires further elaboration. From my understanding, it is non-trivial for an LLM to generate only a single mathematical step with early stopping. If the authors generate an entire solution and truncate it to retain only the subsequent step, this approach would be computationally expensive. 4. In line 125, the authors write *we follow a trade-off rollout strategy between AlphaGo and AlphaGo Zero. Because our tree depth is much shallower than Go games (e.g., a maximum depth of 8).* What is the "trade-off rollout strategy between AlphaGo and AlphaGo Zero"? I don't personally remember such a concept in the AlphaGo work series; and why your tree has a maximum depth of 8? There should have been many solutions with more steps for the MATH dataset. 5. In line 135, *assuming that ...*. Why this assumption is valid? Please elaborate. 6. In Eq. (6), the the loss terms for the value model is minimizing the difference between Q and V. However, in AlphaGo Zero it is minimizing the difference between V and the final reward z (1/-1 indicating the final winner at the termination state). The authors do not provide an explanation of this modification. Please elaborate. 7. In line 164, *This selection is guided by the maximum Q-value*. In AlphaGo and AlphaGo Zero, the selection at inference time is guided by the visited count. Using Q is also a valid choice, but there should be clarification for this modification. An optional ablation study would be better. 8. In Table 3, the SBS with $B_1=2 \text{or} 3$ has shorter average time than $B_1=1$. This looks weird. I guess it comes from the less number of steps. But why $B_1=2 \text{or} 3$ has less number of steps then? Also, the "discussion of majority voting" is really confusing. I still don't understand why they are not comparable after reading it. 9. In Figure 5, where is the Q-value sampled from? For full solution only or also including intermediate of steps? Also why there are three curves in the right figure while there are only two legends? Ref: 1. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. https://arxiv.org/abs/2312.08935 2. Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision. https://arxiv.org/abs/2402.02658 3. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. https://arxiv.org/abs/2406.06592 4. Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training. https://arxiv.org/abs/2309.17179 5. Step-level Value Preference Optimization for Mathematical Reasoning. https://arxiv.org/abs/2406.10858

Rating

4

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors provide a limitations section (A.1); however, it omits several critical aspects: 1. Methods that use the final answer's correctness to infer the reward or value of intermediate steps are prone to false positive and negative issues. Specifically, when the final answer is correct, all intermediate steps are rewarded to some extent. Nevertheless, false positive cases occur when models luckily guess the correct final answer despite incorrect reasoning. Previous related studies, such as Math-Shepherd and OmegaPRM, have acknowledged this limitation. It is recommended that this paper also addresses it in the limitations section. 2. The method relies on an automatic verifier to determine the correctness of the final answer. This verifier is not always applicable to many tasks, such as open-ended text generation (e.g., translation and summarization). The authors did not clearly explain the limitations concerning the scope of tasks.

Summary

This paper leverages the Monte Carlo Tree Search (MCTS) to iteratively train policy and value models by automatically generating process supervision and step-level evaluation signals, eliminating the need for human-annotated process supervision data. Specifically, the method combines the inner capabilities of pre-trained LLMs with the MCTS framework to generate correct and incorrect solution paths and optimize the models based on the node evaluations along these paths. To improve inference efficiency, the paper also proposes a step-level beam search strategy that allows the value model to assist the policy model in navigating more effective reasoning paths rather than relying solely on prior probabilities while avoiding excessive time consumption. Experimental results demonstrate that even without GPT-4 or human-annotated process supervision, this paper' s method performs comparably or better than existing SOTA methods across multiple in-domain and out-domain datasets.

Strengths

1. The work proposes a novel and efficient method that eliminates the need for human process annotations by leveraging the MCTS framework for model self-evolution. 2. The step-level beam search strategy significantly enhances the model's reasoning capabilities while maintaining computational efficiency.

Weaknesses

1. Although the method does not rely on human-annotated process data, it still requires actual answers as reward signals. Future work could explore fully unsupervised approaches. 2. Although the step-level beam search improves efficiency, the MCTS method still has some limitations in terms of computational complexity, which makes its practicality indistinguishable from majority voting.

Questions

N/A

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

N/A

Summary

The paper introduces a novel approach leveraging the Monte Carlo Tree Search (MCTS) framework to generate process supervision and step-level evaluation signals automatically, thus enhancing the mathematical reasoning capabilities of LLMs. This method bypasses the need for costly and labor-intensive manual annotations by allowing the models to train iteratively on both policy and value aspects, with an efficient inference strategy called step-level beam search. This approach demonstrates significant improvements on various datasets, showing comparable or even superior results to existing state-of-the-art methods without relying on external process annotations.

Strengths

The paper leverages a Monte Carlo Tree Search (MCTS) framework to automatically generate both the process supervision and step-level evaluation signals necessary for training LLMs. This approach avoids reliance on human annotated data by generating necessary training signals through interactions within the MCTS environment. As a result, the model achieves comparable or even superior performance to previous state-of-the-art methods on both in-domain and out-of-domain datasets.

Weaknesses

1. Duplicate citations, such as [6], [7] 2. Appendix Figure 9, the reason why the performance of the third round model in MATH at Level 1 and Level 3, Counting & Probability, Number Theory and other categories has declined. Whether it will continue to decline after more rounds. 3. The method integrates CoT and Code for cross-use. How does the performance of the model change if only CoT is used? 4. Evaluate on more out of domain benchmarks, such as GSM8k_hard and GSM8k_robust and OpenLLM Leaderboard 5. How much total training data is used. Explore the effect on other base model, such as Llama2, Mistral 6. Explore the effect of different values of the hyperparameter β in Eq. (8). 7. what is the change in loss of Policy model and Value model during training. 8. What's the accurate of the Value model predictions based on the current State and Action. 9. Insufficient introduction of related work, such as Mathematical Reasoning and MCTS. 10. Differences reasoning ability and solution generation process across different rounds for the policy model , as well as the scoring differences by the Reward Model. 11. How the ground truth label scores for training the first round of the Value Model are obtained, and the procedures for the second and third rounds.

Questions

No question besides the one raised in weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper utilizes Monte Carlo Tree Search (MCTS) to sample high-quality process-supervised data for iterative training, effectively reducing the dependency on GPT-4 and thus lowering the associated costs. Additionally, the authors propose an inference strategy, step-level beam search, which leverages value models to enhance the efficiency and effectiveness of the tree search process.

Strengths

1. This paper is well-written, with a clear introduction to MCTS and the authors' proposed method, step-level beam search (SBS). 2. Their results effectively demonstrate the superiority of SBS over MCTS. 3. The model they trained shows a significant improvement over the baseline model, with an increase in performance from 33.2 to 53.6.

Weaknesses

1. The comparison in the experimental results appears to be rather unfair. Generally, more computation leads to higher performance. In Table 2, the authors compare their fine-tuned model using SBS with greedy decoding for other open-sourced models. In Table 3, they compare SBS with majority voting. It is possible that the improvement stems from the reward model rather than the SBS algorithm itself. The authors should also report the results of best-of-n or weighted majority voting using their value model to demonstrate the effectiveness of SBS. 2. The experimental results are not detailed enough in Tables 2 and 3. The authors should also include major@n with different n values. Perhaps they can plot figures with $B_1$ and $n$ as the x-axis to provide a more comprehensive view.

Questions

1. As mentioned in the weaknesses, what are the results for best-of-n or weighted voting? Additionally, how does the accuracy change as n increases during majority voting? 2. How will the performance change when separating the policy model and value model as two totally different models?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Remain the rating

I've read the authors' responses and am satisfied with the responses to my questions. I've also read the comments from other reviewers. Based on all these, I decide to keep the rating.