SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks

Summary

The paper suggests an approach to solving interactive reasoning and planning tasks inspired by the dual-system theory of human cognition popularized by Kahneman. The approach, SwiftSage, combines a small seq2seq model as System 1 and a LLM as System 2. A heuristic module mitigates between the two. The authors conduct extensive experimentation on a large number of textual reasoning and planning tasks. SwiftSage achieves better performance than previous approaches by a large margin in terms of overall performance, efficiency, and exception handling.

Strengths

The paper tackles an important and well-studied problem of solving interactive reasoning and planning tasks. The authors suggest a nice combination between small seq2seq LMs and LLMs, building on the strengths of each one on its own. While this idea is fairly simple, I consider it a solid contribution. Evaluation is extensive and convincingly demonstrates the superiority of the approach over previous ones in terms of performance, efficiency ad exception handling. Overall the paper is clear to read and well motivated.

Weaknesses

There are some related works that should be mentioned and perhaps even included in the evaluation. One is about combining LLMs and classical planners [1]. The other is about a different LLM-based dual-system approach [2]. Statements about LLMs resembling System 2 in human cognition (lines 50-52) are overblown in my opinion. While LLMs sometimes exhibit a behavior that may seem like deliberate reasoning, some may argue that System 2 in [2] is more akin to a humans’ System 2. I suggest to water down such statements or at least address the controversy. [1] https://escholarship.org/content/qt3qq6w5kx/qt3qq6w5kx_noSplash_2e490248a98936f442c743ad768b65f4.pdf [2] https://openreview.net/forum?id=uyKk_avJ-p4

Questions

Lines 24-26 – are those the only approaches? What about using classical planners as part of the solution (e.g. [1] above). Line 205 – I thought that the two stages of planning+grounding are both part of the Sage module. Here you refer to them as SwiftSage. Heuristic for switching between Swift and Sage. – what are the critical decisions (other than giving the final answer)? Sec. 4.3 – how much of the improvement is attributed to a balanced training set? Has this been checked on other models?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Limitations are adequately addressed in the paper.

Summary

The paper proposes SwiftSage that predicts a high-level plan by Swift and prompts LLMs to conduct each high-level subgoal by Sage. Swift is trained by imitation learning for action planning as fast intuitive thinking and Sage utilizes LLMs for better generalizable planning and exception handling as deliberate thought processes. The proposed method achieves strong performances on the ScienceWorld benchmark over the recent state-of-the-art.

Strengths

- The paper is generally written well and easy to follow. - Utilizing LLMs for exception handling seems reasonable. - The overall two-staged framework (intuitive planning -> deliberate planning) is intuitive and sounds sensible. - The proposed method achieves strong performance over prior work by large margins in most cases in the ScienceWorld benchmark.

Weaknesses

- Swift adopts imitation learning to train an encoder-decoder (T5) model but I'm not sure why not just prompting LLMs (see Q1). - The environment provides if the agent fails at an action, which might not be always available (see Q2). - SwiftSage utilizes LLMs for exception handling but it is not clear if strategies for exception handling provided by LLMs are always useful and can be conducted by agents (Q3). - The motivation of each condition of switching to Sage is missing (see Q4).

Questions

- Q1: The Swift module fine-tunes a T5 model with oracle trajectories of training tasks. But there is some work [14,A] that plans a sequence of actions by providing task-relevant examples to LLMs. Is there any reason why the authors take imitation learning rather than prompting LLMs? Although it is fine-tuning and this is quite efficient compared to training from scratch, it does require additional training. In addition, wouldn't it be that planning with LLMs can generalize better than the imitation-learning models as we can leverage commonsense knowledge of LLMs? - Q2: It seems that SwiftSage can perfectly recognize when and why it fails based on the feedback from the environment, which might enable Sage to address exceptions effectively. Can the proposed method address cases where the failure feedbacks are incorrect? - Q3: I'm not sure the proposed method can ensure that LLMs always provide helpful exception-handling strategies. If LLMs provide some strategies that the agent cannot conduct (e.g., due to the absence of objects), how can the agent address the exception? Note that this is not about the excitability of plans, which is addressed in the "Grounding stage" in Sec. 3.3. - Q4: In Sec. 3.4, it seems missing the motivation of the conditions to switch to Sage. Why does each condition imply when to switch to Sage? It would solidify the switching conditions to include them. References\ [A] Liang et al. Code as Policies: Language model programs for embodied control. ICRA, 2023.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations.

Summary

This paper proposes a dual-module approach for ScienceWorld which combines an imitation learning agent based on t5, with an llm-based module which activates based on a heuristic to re-plan its actions, using a buffer of actions to be executed. The approach is novel, sensible and well-motivated, drawing from the concept of slow-fast thinking from psychology. The results significantly advance the state of the art on ScienceWorld, and significantly outperform the compared methods which include state of the art LLM methods like Reflexion.

Strengths

- The paper significantly advances the state of the art on a difficult and interesting task involving language-based interaction, ScienceWorld. I expect that the proposed model will be a useful benchmark for many tasks of a similar type. - The method is sound and well designed. Using seq2seq for a history-aware imitation learning is a sensible model for "fast thinking". And using an LLM for "deliberate thinking", i.e. object and progress tracking, planning and exception handling is also reasonable. Both components are well crafted, and they are nicely integrated in a complementary manner. - The ablations are informative - Ablations show that the Swift module alone establishes a new state of the art, while GPT-4 almost doubles this performance. However chatgpt-turbo, which is comparatively much weaker than GPT-4, also improves Swift-only by about 25%, which is encouraging.

Weaknesses

- I'm curious about the sensitivity of the method to prompt engineering. Q1~5 are quite comprehensive and capture a lot of the core aspects required to successfully solve the game. This is possible for ScienceWorld but would not be possible for many other tasks. Would it be possible to ablate the Qs to show prompt sensitivity? - A weakness of the model is that it relies on a heuristic to decide when to activate the Sage module. I'm curious whether there are no better alternatives or more general solutions to the problem of integrating the two modules?

Questions

See above.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitations discussion is adequate.

Summary

This paper proposes a new agent framework for action planning in complex interactive reasoning tasks. The authors leveraged an imitation-learning trained "light-weight" LLM for fast inference and an LLM (e.g. GPT4) for slow reasoning with proper designs of prompts and interaction between fast/slow modules. The proposed model achieve state-of-the-art results on ScienceWorld, outperforming existing models by a large margin.

Strengths

- This idea of two-system in LLM is a new agent framework that could potentially be extended to many more settings and tasks. - The proposed model outperforms existing baselines by a large margin on ScienceWorld which proves the effectiveness of the two-stage design.

Weaknesses

- One concern about the current framework is its generalizability to more common tasks. It seems that the current design is strongly related to the ScienceWorld benchmark, it would be best to see if the current prompt/interaction designs could be still effective in customized settings, how to define the prompts for a general task, and also how sensitive they are to for example customized defined states, scores, etc. - As 770M is already quite a big model before the LLM era, is the current Swift module capable of what we desire of accurate fast reasoning? This is reflected by the swift-only experiments that can already outperform existing RL/IL-based learning methods by a large margin. It makes one wonder to what extent can swift-module handle the reasoning problem, what are its failure cases, when does it need the Large LLM for slow reasoning.

Questions

See the Weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have properly addressed the limitations of their work.

Rebuttal Acknowledgment

I thank the authors for addressing my concerns with a comprehensive explanation. The thorough response has clarified the issues I raised. I am happy to raise my rating accordingly.