Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing

Summary

This paper introduces ALPHALLM, which is an imagination-searching-criticizing framework designed for self-improvement. Inspired by AlphaGo, authors integrate MCTS and LLMs to establish the self-improvement loop. Additionally, authors proposed eta-MCTS, which is a decoding method used to reduce the search space. The experiments show that ALPHALLM can increase LLMs’ reasoning ability obviously, especially combined with eta-MCTS decoding.

Strengths

1. It’s innovative to use tree search to do self-improvement and it’s good to identify some challenges very clearly. 2. The experiments show that the reasoning ability of LLMs can be improved effectively.

Weaknesses

1. It could be more clear in some parts and including some examples would be helpful. For example, in the part of 4.2 data synthesizing, I’m curious about how the synthesized data looks like. 2. Authors identify that one of the challenges in working on LLMs’ self-improvement is the difficulty to get clear evaluations, and they proposed some method to get the evaluations, i.e. value function, PRM, ORM. But they do not discuss whether these methods are reliable to get perfect feedback.

Questions

Please refer to the part of weaknesses.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are sufficiently discussed.

Summary

This paper proposes an imagination-searching-criticizing approach called ALPHALLM to enhance the capabilities of large language models (LLMs). ALPHALLM integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving framework.

Strengths

- This paper introduces a novel approach to LLM self-improvement using MCTS, presenting an interesting concept. - The paper is well-written, and the proposed imaginative-searching-criticizing approach is clearly explained. - Using only 7.5k/3k final math answer annotations and after just two iterations of self-improvement, ALPHALLM achieves impressive results: 92.0 on GSM8K and 51.0 on MATH. These results are remarkable.

Weaknesses

- Given only final math answer annotations, ALPHALLM essentially performs the final-label (reward) classification problem and tree search based on the predicted reward to generate rationales for each final answer. It is still unclear how ALPHALLM could outperform other LLM-SFTs that utilize both rationale annotations and final answer annotations. - ALPHALLM is a general framework. The authors should apply ALPHALLM to other tasks where learning signals are clear to demonstrate the overall effectiveness of this self-improvement framework.

Questions

- Have you ever tried using your value/reward function to perform RL fine-tuning with PPO?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

None.

Summary

The paper proposes AlphaLLM, a tree-search enhanced framework with a few improvements over Data Synthesizing, option-level MCTS, Importance-Based Adaptive Branching, state merging, fast rollout, critic function and policy improvement process. Experimental results verify the framework's effectiveness on GSM8k and MATH.

Strengths

1. The technical contribution seems solid. The paper proposes a comprehensive framework and address modifications/improvements over the full pipeline of tree-search. 2. Experimental results demonstrate great potentials for the proposed algorithms.

Weaknesses

1. The writing is not clear enough. It is not that clear to me how the option-level is implemented to separate sentences (how the Termination function is determined). 2. There are many different components in the pipeline design but the ablation studies are not enough to validate them one by one. For example, It at least need to involve: 2.1 Comparison with other heuristic search methods, for example: (1) the beam-search + value function described in [1, 2]) (2) majority-vote or reranking with ORM. 2.2 Wall time or token consumption comparison between different methods. 2.3 The ablation studies seem to be too simple to present how different components can really influence the performance. Pure with/without results seem to be too simple. More experiments like how to control the hyperparameter for these things are required. For example, how different heuristic functions can influence the State Merge, how different fast rollout model can influence the effiency, how the thresold in ORM in the process of data generation can influence the improvement process, etc. Reference [1] Feng, Xidong, et al. "Alphazero-like tree-search can guide large language model decoding and training." arXiv preprint arXiv:2309.17179 (2023). [2] Yu, Fei, Anningzhe Gao, and Benyou Wang. "Outcome-supervised verifiers for planning in mathematical reasoning." arXiv preprint arXiv:2311.09724 (2023).

Questions

1. What is the difference between value function training and PRM training? From the description it seems that the PRM's training is exactly the same as value function training in Monte-Carlo estimation (except you tend to take more simulations starting from a fixed state). Also, it seems that the technique described in the critic section has almost been described in many previous works: value function training in [1], PRM training in [2, 3], and ORM in [4], the author needs to clarify more on their own contributions and make proper citations. 2. One difference between your work and [1] is that they leverage the value function itself to backward (like AlphaZero) while you are using the fast model to rollout to reach the terminal state, which is like the initial version of AlphaGo (where they have a fast-move network) and [5]. What do you think of these differences and why do you make this design choices? Reference [1] Feng, Xidong, et al. "Alphazero-like tree-search can guide large language model decoding and training." arXiv preprint arXiv:2309.17179 (2023). [2] Wang, Peiyi, et al. "Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning." arXiv preprint arXiv:2312.08935 (2023). [3] Lightman, Hunter, et al. "Let's verify step by step." arXiv preprint arXiv:2305.20050 (2023). [4] Uesato, Jonathan, et al. "Solving math word problems with process-and outcome-based feedback." arXiv preprint arXiv:2211.14275 (2022). [5] Hao, Shibo, et al. "Reasoning with language model is planning with world model." arXiv preprint arXiv:2305.14992 (2023).

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

See questions and weaknesses.

Summary

The paper introduces a method for self-imporvement of LLMs called AlphaLLM. The method consists of three components: Generation of expert trajectories Effective Monte-Carlo-Tree-Search over the LLM outputs (nablda-MCTS) A series of critics for evaluating reliable reward signal (value function, Process Reward Model, Outcome Reward Models) The method can be used as a rollout mechanism to generate high quality outputs. To help deal with the large branching factor, the authors propose three subcomponents: State space is set to Options, which is neither token space nor sentence space, but a variable length sequence which terminates based on an auxiliary heuristic beta. States are merged using another auxiliary function p_vM Fast rollouts are used with a smaller LLM to help estimate. The authors then demonstrate incredibly strong empirical results on both MATHS and GSM8K. Training with the proposed method on LLAMA-2-70b is able to achieve 51% on the MATH dataset.

Strengths

The method seems to work well! The discussion on state merge and options helps express difficulties in applying MCTS. The paper seems to combine many subcomponents to produce a strong algorithm. The branching factor is well-motivated.

Weaknesses

Lots of experimental details are missing. There is very little discussion of how AlphaLLM is trained or what base model it is (it's only mentioned in the introduction!). In particular, no information is given about the actual training of either the policy (LLM), the Critics, or the heuristic functions p_vM. The explanation of the options is lacking - even looking at A.2 the explanation of options is unclear. Is Beta learnt as a function or generated in some other way? Are the set of options I, learnt before a rollout or generated on the fly? Ablations are missing - in particular using different state methods, removing the Importance-Based Adaptive Branching, State Merge or Option-Level MCTS. The prompts in the appendix are reduced so we can’t replicate. No Code provided. Weird choice of baselines - surely to keep evaluations consistent with other LLMs, you should allow models to generate the same amount of output tokens with access to the PRM or ORM? I find it hard to believe COT on Claude-2 is a fair comparison, when the amount of optimisation pressure applied to multiple components of the method (including gradient updates). There is no example of a rollout provided.

Questions

- What policy was used in the end for the fast rollout? - The prompt templates are missing key parts - [A detailed rubric that specifies how to evaluate a step of a task] would be helpful if shared. - In particular when comparing to GPT-4 or other model perform was an equally informative prompt used? (Or at least a prompt of equal input tokens). - If the synthetic data generated is used few-shot by llama2, how does your model perform? - I think the related work is missing some key papers such as: [https://arxiv.org/abs/2203.14465] (the STAR family of papers)

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The paper in its current form is not replicable.