Summary
The paper proposes an architecture towards autonomous agent (SALA), which use one large language model (LLM) to generate thoughts, actions, and self-adaptations by concatenating two Reflexion (Shinn et al., 2023) exemplars and two ReAct (Yao et al., 2023) exemplars for each task. They show that this approach is more effective than ReAct on the twelve selected tasks from ALFWorld environment and it achieves a success rate of 83%. They also find that the self-adaptation is particularly important for decision making in the next trial.
Strengths
**Originality**: The paper modifies the Reflexion that uses two large language models (LLMs) to use only one LLM to generate thoughts, actions, and self-adaptations, which may use less memory. However, there are problems in the originality, which will be further discussed in the Weakness Section.
**Quality**: (Experiments are not sufficient, which will be detailed in Weaknesses Section)
**Clarity**: The paper is easy to follow. Illustrative figures are helpful for understanding the paper.
**Significance**: (The proposed method SALA lacks significance, which will be detailed in Weaknesses Section)
Weaknesses
**Lack of Novelty**
- The proposed method SALA use one model for action generation and self-adaptation, but this lacks novelty in the methodology because Reflexion pipeline is sequential and one can use one LLM for different tasks with different prompts.
- Minor modification from the original method Reflexion is not considered to be novel if there's no superior results to the original method. The paper does not compare the method SALA to Reflexion on either accuracy or efficiency.
**Lack of Experiments**
- The experiments only compare the proposed method SALA with ReAct, which is less capable than Reflexion (the method augmenting the ReAct with self-reflection). Since SALA applies minor modification from the Reflexion, authors should add experiments with the Reflexion.
- There are only text-based game ALFWorld in the experiments, which is not enough to demonstrate the advantages of the proposed method. Authors should consider adding other datasets used in Reflexion like HotpotQA [1] for reasoning, and MBPP [2], HumanEval [2], LeetcodeHardGym [3] for programming.
- The paper only have results on the open-sourced language model gemma-2-9b-it, which are not enough. Authors should evaluate at least 3 open-sourced models with different model size (e.g. 30B, 65B). Additionally, it is recommend to add API models like GPT-4o since they are generally more capable than those open-sourced models which may lead to different model behaviors with different methods. At the end, authors should aggregate the results and also average them to provide a more comprehensive evaluation.
*References*
[1] Yang, Zhilin, et al. "HotpotQA: A dataset for diverse, explainable multi-hop question answering." arXiv preprint arXiv:1809.09600 (2018).
[2] Austin, Jacob, et al. "Program synthesis with large language models." arXiv preprint arXiv:2108.07732 (2021).
[3] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).
[4] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
Questions
**Questions**
- Reflexion uses three models: an Actor to generate text and actions, an Evaluator to score the outputs from the Actor and a Self-Reflection model to generate verbal reinforcement cues for self-improvement. Why the paper states that Reflexion uses only two models? The paper needs more clarifications on this discrepancy.
- What is the main motivations for modifying the Reflexion to use a single LLM to generate thought, action, and self-adaptation? Why this minor modification would differentiate this work from Reflexion? Authors should clarify the motivations.
**Suggestions**
- Authors should add a column of average success rate and average steps in the table for easier comparison (line 327).