Recursive Introspection: Teaching Language Model Agents How to Self-Improve

Summary

This paper proposes a general paradigm for fine-tuning LLMs such that the LLM can iteratively refine from its previous outputs in an in-context learning fashion. Experiments are conducted on math tasks where the learned model can self-improve with multi-turn outputs.

Strengths

1. The proposed RISE framework is novel and general. The overall framework can be possibly extended to a wide spectrum of tasks. Although the current form of the method remains simple, it can serve as a good starting point for learning self-adaptive LLMs. 2. The experiments are solid with a diverse collection of benchmarks as well as baselines including GLORE and Self-Refine. The ablation studies and analysis are thorough. 3. The paper is well-written and easy to follow.

Weaknesses

There are no major weaknesses in this paper. But there may be some potential directions for further improvement. 1. The current learning method fundamentally remains a distillation process, either from a teacher or from the base model itself. Is it possible to directly perform RL over the trajectories? Note that this is essentially an in-context RL process. It would be interesting to see whether this can be achieved. 2. The experiments are primarily conducted on Math benchmarks. The paper can be even stronger if other tasks can be considered. 3. The authors list two possible reasons why having an improving trajectory can lead to better final outputs. I fully agree with the two reasons. It can be more convincing if some concrete examples supporting these two hypotheses can be discussed. 4. RISE looks so much like in-context RL[1,2] because in-context RL also aims to learn a model that you can put trajectories in the context window and then the LLM can run RL to improve the output. Some discussions on this can be appreciated. [1] In-context Reinforcement Learning with Algorithm Distillation, M. Laskin et al., https://arxiv.org/abs/2210.14215 [2] Supervised Pretraining Can Learn In-Context Reinforcement Learning, J. Lee et al., https://arxiv.org/abs/2306.14892

Questions

See comments above.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations have been well addressed in the paper.

Summary

This paper introduces RISE: Recursive IntroSpEction, a novel approach to fine-tuning Large Language Models (LLMs) for self-improvement. The core idea is to enable LLMs to introspect, reason, and correct their mistakes over multiple turns. This is achieved by treating the fine-tuning process as a multi-turn Markov decision process (MDP), where the initial state is the prompt. The RISE algorithm iteratively fine-tunes the model using a combination of on-policy rollouts and reward-weighted regression (RWR) objectives. The method is shown to significantly improve the performance of 7B Llama2 and Mistral models on math reasoning tasks, outperforming several single-turn strategies.

Strengths

- This paper presents a novel method for enabling self-improvement in LLMs, addressing a significant limitation in current models. - The use of iterative fine-tuning and on-policy rollouts is a robust approach to teaching models how to correct their own mistakes. - The approach is designed to be general, enabling self-improvement across a diverse set of problems and scenarios.

Weaknesses

- While the results on GSM8K and MATH are promising, additional experiments on other types of tasks (e.g., natural language understanding, code generation) could strengthen the paper. - The success of RISE may depend heavily on the initial quality of the model. Models that are already strong may benefit more from this approach than weaker models.

Questions

1. In Section 4.2, it is mentioned that “(3) it must not contain any rollout that degrades in a subsequent turn. Our data collection strategy that satisfies these desiderata.” How is performance degradation determined? Does it require manual judgment? 2. Does “starting from a boosted model” introduce unfairness? Because the article does not mention whether other baselines undergo SFT (or maybe I missed it). 3. In Chapter 7, it is mentioned that “RISE requires running manual iterations.” What does "manual" mainly refer to here? 4. How should Figure 2 be interpreted? Why is the Success rate for Iteration2 Model lower than the Boost Model? 5. In Appendix C.2, it is mentioned “To control the randomness,” however, setting temperature to 1.0 and top_p to 1.0 seems counterintuitive? 6. How does the model locate errors in Figure 11? “the model is able to locate the error and perform local computation correctly.” 7. In Figure 4, what does on-policy+Expert (best of n) mean? The article mentions “Best-of-N” means using the best sample out of N from the learner (here N = 16), which seems to conflict with expert? Or can the learner here refer to the expert? And what does Expert+Expert mean? Is it related to the “DAgger [34]”-style approach? 8. If the reward function is a sparse binary indicator (0 or 1), how does it reflect the superiority of Reward-weighted RL? What is the difference between it and “simply imitating filtered successful data”? 9. Typos: figure 1 right, turn 1 response -> turn 2 response.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The author has already discussed the limitations ofthe method and its potential impacts in the paper, and provided possible solutions.

Summary

The manuscript tries to solve the problem that existing large language models (LLMs) dono't have the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. The authors propose a fine-tune approach, so-called RISE (Recursive IntroSpEction), that aims to enhance the self-improvement capabilities of LLMs to tackle these complex problems. The RISE method refines LLMs through an iterative fine-tuning process that teaches them to improve their responses following additional environmental feedback. It reformulates the single-turn fine-tuning problem as a multi-turn Markov decision process (MDP), and trains using a reward-weighted regression (RWR) objective. During inference, RISE enables models to detect and correct their previous mistakes over multiple iterations, thereby incrementally enhancing their problem-solving abilities. The results show that LLMs trained via RISE can produce correct responses on more prompts, improving over turns for more challenging prompts.

Strengths

The RISE algorithm, which leverages principles from online imitation learning to improve the LLM's capacity for self-improvement over multiple turns of interaction, and queries expert supervision on states attained by on-policy rollouts. It poses fine-tuning for a single-turn problem as solving a multi-turn Markov decision process (MDP), where the initial state is the prompt. The learner is fine-tuned using a reward-weighted regression objective, which is able to learn from both high- and low-quality parts of rollouts. Experimental results demonstrate that, RISE can enable similarly-sized LLMs to with self-improvement capabilities, with a notable advancement in monotonically increasing task performance in mathematical reasoning tasks.

Weaknesses

1. In the MDP formulation, the action is denoted as the output response of the foundation models. This means that the action space would be very large, which make the MDP hard to learn. 2. Three conditions in the data collection, need a lot of sequentially prompts or human feedbacks. In the experiments, the scale or number of iterations is somehow small, corresponding to the self-improvement problem. 3. The equation 4.6 indicates that the RISE is actually a sequentially weighted supervised learning, and the RL/MDP model is confused with the original LLMs.

Questions

1. How many training data is collected and used? 2. The reward is 1 if and only if a = y*. How to judge the reward when collecting data, when the LLM give an improving answer but still not right one? 3. In equation 4.4-4.6, the LLM is treated as a offline RL Model with weighted reward genereted. It might be better to provide more theoretical details.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Plenty of experimental results are presented. However, there are some computational constraints, that all the iterations is less than 3, and the turn is fixed as 5. The scale or number of iterations in the experiment is somehow small, corresponding to the self-improvement problem.

Thanks

The new results look great to me!

Discussion Period

The discussion period is almost over, so both authors and reviewers please respond to any unaddressed questions. Reviewers, be sure that you have all of the information you need from the authors, since after the 13th, they won't be able to respond.