Reflexion: Language Agents with Verbal Reinforcement Learning

Summary

This paper presents a method by which LLMs can improve task performance through a form of learning in language: distilling unsuccessful attempts down to a "lesson learned" via a self-reflection process. The work shows that this idea leads to improved performance within a variety of task domains, and includes fairly thorough ablations, which I appreciated.

Strengths

I find this to be a fairly satisfying paper overall. * The idea is compelling, and well executed. * The experiments demonstrate that the approach is beneficial across three fairly diverse LM application domains, which shows that it is not simply a narrow improvement. * The ablations are fairly thorough, and make the results more compelling. * The paper is generally well-written and presented.

Weaknesses

I have a few suggestions on how the paper could be improved: 1) I think this work could be more thoroughly integrated with the existing literature on learning from language feedback, both within LLMs (e.g. https://arxiv.org/abs/2204.14146 and https://arxiv.org/abs/2303.16749) and RL agents (https://proceedings.neurips.cc/paper_files/paper/2022/hash/0113ef4642264adc2e6924a3cbbdf532-Abstract-Conference.html and https://arxiv.org/abs/2112.03753). Another recent paper https://arxiv.org/abs/2304.03442 proposed a similar reflection approach, though with different motivation; it came out quite recently though so there is no expectation to refer to it. 2) Evaluations on GPT-4 are challenging to interpret. We don't know what data the model was trained or tuned on, and even if the datasets used were generated after the pre-training cutoff, the model may have encountered them in the fine-tuning or RLHF stages. Indeed, OpenAI explicitly uses information submitted to their chat (at least) as training data; can we guarantee that no LeetCoder submitted the problems to GPT-4 to check their solutions? This problem is exacerbated if the benchmark has previously been evaluated against GPT-4, as it seems these problems have; there have been substantial concerns about GPT-4's coding performance evaluations being affected by data contamination. It would be best if the authors at least acknowledge these issues in the manuscript; ideally, they would also evaluate against an open language model for which the training process is more transparent, although undoubtedly such a model might not perform as well.

Questions

See above for some suggestions.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are discussed

Summary

The paper introduces a novel framework, Reflexion, to improve the learning capabilities of language agents. Instead of traditional reinforcement learning methods, Reflexion uses linguistic feedback, where agents verbally reflect on task feedback and store these reflections in an episodic memory buffer to enhance decision-making. The authors demonstrate that Reflexion, which can incorporate various types and sources of feedback, outperforms baseline agents across diverse tasks, including text games, natural language question-answering, and code generation.

Strengths

1. The paper introduces a unique framework, Reflexion, that uses linguistic feedback for reinforcement learning in language agents. This is a significant departure from traditional methods that rely on weight updates. 2. The paper offers a good explanation of their proposed methods and is easy to follow. Good visualizations are also included to help with reader understanding. 3. Abundant experiments have been conducted across 3 different tasks, covering text games, question answering, and code generation. A comprehensive analysis has also shown the effectiveness of the reflexion method.

Weaknesses

1. Although reflexion shows significant improvements over baselines on three different tasks, I still feel that reflexion shows limited generalization ability on different tasks. The key modules in the framework, including evaluator and self-reflection, have varying designs for different tasks. In other words, the specific design details may only work on one task and may fail on the other ones. 2. The results for comparison in the question-answering task seem to be unfair. The reflexion framework leverages the environment binary reward of whether the answer is correct as the ground-truth answer or not. This leaks a little information from testing question-answer pairs. At least the ReAct baseline did not include such information. 3. There are some related works mentioned in Section 2. However, there is no comparison between Reflexion and these methods in the experiment sections.

Questions

1. Can you offer some statistics on the running time of the reflexion framework? As it seems that the optimization process is time-consuming. 2. I am very curious about the comparison between reflexion and methods like self-refine and self-debugging. 3. Can you offer some detailed analysis of the differences between your method (reflexion) and simple ReAct+self-refine? 4. Is the quality of test cases generated in code generation tasks important? Are they guaranteed to be correct?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately introduced the limitations of their work.

Summary

This paper proposes a novel framework to do RL style learning through LLM. The idea is interesting. The authors have also done evaluation to show this technique/framework can work better than baseline methods w/ a few different tasks.

Strengths

The author proposed a novel framework to let LLM learn new tasks through trial and error. The framework seems general and can handle multiple tasks.

Weaknesses

1. While the framework is general. The devil is in the details. It seems for each specific task, it requires different heuristic and tuning. Also different task may require different LLM/VLM as actor/evaluator/self-reflection. 2. In terms of ablation, since this paper focuses on the framework, it seems more useful to ablate different framework level choices. i.e. is the self-reflection module really the key component in this framework, or can we bypass it by some other way, such as prompting?

Questions

Will this framework also work well w/ VLMs? (if we also have vision/images as input)

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

See weakness section.

Summary

The paper presents Reflexion, a framework to reinforce language agents by using language feedback. Reflexion agents verbally reflect on task feedback signals, then store their own reflective text in a memory buffer to improve their decision-making in future trials. Reflexion is flexible and effective across diverse tasks (sequential decision-making, coding, language reasoning). The paper also explores different types and sources of feedback signals, feedback incorporation methods, and agent types, and provides insights into how they affect performance.

Strengths

1. This paper proposes Reflexion, a new framework for “verbal” reinforcement that conditions a policy on the agent’s “memory” which is beneficial for the development of LLMs, reinforcement learning, reasoning, code generation, and so on. 2. The results across diverse tasks including sequential decision-making, language reasoning, and programming are very promising with a large advantage. 3. This paper proposes a new benchmark, LeetcodeHardGym, consisting of 40 challenging Leetcode questions in 19 different programming languages, which is beneficial for the program synthesis and code generation community.

Weaknesses

1. Various LLMs (the GPT family, the vicuna family, and models with different sizes) should be evaluated to show reflexion’s generalization ability across models. Although reflexion performs well, it is important to analyze its border. 2. Ablation study about long/short term memory. For example, in ALFWorld the baseline is react which resets the environment and starts a new trial whenever self-reflexion is suggested. I take it as “reflexion w/o long/short term memory”. Then, how about the performance of “reflexion w/o long-term memory (but w/ short-term memory)” and the performance of “reflexion w/o short-term memory (but w/ long-term memory)”?

Questions

1. According to line 117, the evaluator is not necessarily an LLM, right? But it is pictured as “LM” in figure 2 which is a little bit confusing. 2. In algorithm 1, $\pi_\theta$ is not relevant to trajectory $\tau_{t-1}$, the short-term memory. But in figure 2 (a) and section 3, the Actor receives short-term memory as its input. 3. How does short-term memory work on programming tasks? Is reflexion = self-debugging + CodeT?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The author has discussed the limitations well.

Summary

The paper introduces Reflexion, a framework for in-context reasoning and decision-making for LMs. It allows the LM to iteratively attempt to solve a task and then reason about the results of its attempts and plan improved courses of action. The authors evaluate Reflexion on a diverse array of tasks (navigating and acting in text environments, question answering, code generation) and report impressive results, in some cases establishing new state-of-the-art. They also compare Reflexion to useful baselines and conduct instructive ablations.

Strengths

1. The paper proposes a sophisticated planning algorithm (composed of short-term and long-term-memory, gating mechanisms and feedback simulators) that can be run (almost) entirely in-context by an LM. I think this approach is very powerful and this paper does a good job showing its promise. 2. The paper is well-written and easy to follow. I like the error analysis paragraph accompanying each section. 3. I appreciate effort put into thorough benchmarking of code generation capabilities, including translating MBPP and HumanEval to Rust and introducing LeetcodeHardGym that prevents tests cases from having been leaked into GPT-4 data. 4. The empirical results obtained by Reflexion are impressive, especially pass@1 for HumanEval.

Weaknesses

1. It would be useful to compare Reflexion to baselines also in terms of inference efficiency. For instance, how many tokens are typically needed for a Reflexion run (vs e.g. ReAct or CoT) to reach an answer in a given task? Is the performance Reflexion bottlenecked by context window size and could we expect it to improve as context windows of LMs increase? 2. Similarly, I am a bit disappointed that authors don’t report any error bars. I understand inference costs, but the authors could at least compute errors bars based on variance across problems (in code generation) or across tasks (in AlfWorld and HotPotQA). Alternatively, I’d rather see a less diverse task array than no error bars. 3. Authors frequently compare Reflexion to human cognition (e.g., “This is akin to how humans iteratively learn to accomplish complex tasks”, line 31-32; “similar to the way that humans remember fine-grain recent details while also recalling distilled important”, lines 141-143). I think these comparisons are overblown, not supported by cognitive science citations and potentially misleading. For instance, there are numerous dissimilarities between the kind of memory being used in Reflexion and human memory and the topic deserves a more careful discussion that just highlighting superficial similarities in passing.

Questions

1. It's not entirely clear to me which parts of the algorithm happen within the "main" context window of Reflexion and which are encapsulated into separate, local context windows that are then discarded (my understanding is that this is the case, e.g., for unit test generation). I'd appreciate if the authors could make that more clear. 2. How does Reflexion scale (down) with model size? Are GPT-4-level capabilities necessary for it to yield improvements or does it also work with `gpt3.5-turbo` and `text-davinci-003`? 3. How was the error analysis in section 4.1 conducted, i.e. who classified failures as being due hallucination or due to insufficient planning?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Certain limitations and societal impacts are discussed. I think it could be good to complement that with a discussion of risks associated with making LMs prompted to be agents act autonomously by allowing them to set their own subgoals and to plan with rich world models (e.g., Perez et al., 2022; Ngo et al., 2023). This could have unknown failure modes if those agents are given unrestricted Internet access and are not supervised. Perez et al., 2022. Discovering Language Model Behaviors with Model-Written Evaluations Ngo et al., 2023. The alignment problem from a deep learning perspective

Thanks for the response

Thanks to the authors for the response! To clarify my comment on contamination: while I acknowledge that the new benchmark was created after the pretraining cutoff, we know that GPT-4's training has continued up to the present (at the very least to address vulnerabilities and introduce new features); it's quite reasonable to presume they use user chats for that training, and so it is still possible the model has been trained on these problems. The fact that it does better with Reflexion than alone is promising in that regard, but does not completely rule out the possibility (just because an LM has been trained on something does not mean it will always get it right). I still think it might be worth acknowledging this more explicitly, even if it is a general problem. However, I do want to reiterate that I think this is a valuable work that should certainly be accepted.

Thanks for Responses, and Look Forward to Discussions

We want to thank Reviewers k1Ks, V8mX, 6XLB for responding to our rebuttal, and we look forward to engaging in discussions with Reviewers BKdQ, U48e, to see if their concerns are addressed, and address any potential remaining concerns in the final 4 days. Thanks in advance!

Thanks for the Response

I think most of my concerns have been addressed by the authors. Thanks for the efforts during the rebuttal. Although some of the points I listed in the review seem to be framework limitations, I still find the paper interesting and useful. I will maintain my positive score.