AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning

Summary

This paper introduces AutoManual, a framework enabling LLM agents to autonomously build understanding of new environments and generate instruction manuals through interactive learning. The system comprises three main components: a Planner agent generating code-based plans for environment interaction, a Builder agent updating a structured rule system based on observed trajectories, and a Formulator agent compiling rules into a comprehensive manual. To mitigate hallucinations in rule management, the authors implement a case-conditioned prompting strategy for the Builder. The framework is evaluated on the ALFWorld and MiniWoB++ benchmarks, demonstrating high success rates. AutoManual aims to address limitations of prior approaches such as path dependency and reliance on extensive human-provided instructions, potentially improving the adaptability and generalization of LLM agents in diverse environments.

Strengths

Originality: - Novel and creative framework for online rule learning and manual generation by LLM agents - Innovative case-conditioned prompting strategy for rule management Quality: - Detailed examples and case studies demonstrating the approach - Careful comparison to relevant baselines (e.g., ReAct, Reflexion, ExpeL) Clarity: - Clear explanation of the overall framework and key components - Well-structured paper and easy to follow - Detailed appendix with prompts, examples, and implementation details Significance: - Addresses an important challenge in building more general and adaptable LLM agents - Demonstrates a path towards more autonomous learning in new environments

Weaknesses

1. Heavy reliance on very capable LLMs like GPT-4, and dependency on oracle feedback during rule construction process limits broader applicability, do we have results by fine-tuning smaller models on the generated data? 2. Scalability to larger, more complex environments is unclear - the current approach puts all rules in LLM context. It’s not clear how easily the method can be generalized to other environments, for example, SWE-Bench. 3. Limited evaluation of the quality and usefulness of the generated manuals for humans. Lack of theoretical analysis on the rule learning process.

Questions

1. How is the initial rules and examples selected and how sensitive is the performance to the quality of the initial rules/examples provided? How this initialization needs to be changed when we switch to different environments? 2. In Section 3.2 Skill and Reflection library construction, the most similar task is retrieved from the skill library when new task comes, how is the similarity between task been defined? 3. In 3.3 Rule types definition, how does the 6 rule types come up? 4. Table 2 shows the success rate of LLM agents on 9 task types with feedback, what are those feedback, and is the feedback rely on an oracle who know the correct answer/solution for the given query?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors provide a reasonable discussion of limitations, including reliance on GPT-4 and potential scaling challenges. However, more concrete mitigation strategies are missing. They also touch on broader impacts, noting both positive potential (e.g., improved safety, worker training) and risks (e.g., unpredictable behaviors). Side note: should the limitation and broader impact discussions be included within the main 9-page scope rather than in the Appendix?

Summary

LLM agents show promise in many environments but require expert prompts in specific environments to perform well. This paper introduces the AutoManual framework which autonomously builds a 'manual' of rules through interactions to help adapt to new environments. The framework consists of 3 agents: the Planner which generates code to carry out a plan, the Builder which updates rules, and a Formulator which consolidates the rules into a manual. The Builder updates rules through a original case-conditioned prompting strategy to mitigate hallucinations. AutoManual is evaluated on 2 benchmarks: AlfWorld and MiniWoB++ and achieves high performance.

Strengths

$\textbf{Originality}$: AutoManual breaks apart the knowledge acquisition problem into 3 agents, one of which uses a case conditioned prompting strategy, which breaks down rule updates into a decision tree that simplifies LLM responses, mitigating hallucinations. $\textbf{Quality}$: AutoManual is evaluated on 2 benchmarks against popular baselines and related knowledge acquisition works and provides insightful ablations into its components. High quality. $\textbf{Clarity}$: Plenty of helpful figures and very clear approach. $\textbf{Significance}$: Autonomously collecting knowledge to formulate an instruction manual simplifies human efforts when developing prompts for LLM policy approaches.

Weaknesses

This paper is pretty strong already so there are only 2 things I wanted to point out that could make it stronger. 1. The ablations are very helpful in justifying design decisions but the table looks odd with seemingly random kinds of ablations. I understand if there is a concern with resource-constraints running all 16 combinations of ablations but why this set of 7 ablations? 2. Temperature 0 for OpenAI is typically a red flag to me that performance is brittle since there are no guarantees of determinism [1]. Despite experiments being run 3 times and averaged, I've experienced varying performance across different days. I'd be interested to see if a subset of the results remains relatively the same. [1] https://community.openai.com/t/why-the-api-output-is-inconsistent-even-after-the-temperature-is-set-to-0/329541

Questions

Suggestions - Provide an explanation/justification for the incomplete ablation table in the paper - Provide a subset of results (e.g. a single column) to verify performance isn't brittle due to temperature 0 Questions - Manuals are constructed by sampling specific task types in both environments. I am curious whether you experimented with constructing manuals with limited task types and testing on unseen task types. In other words, can the manuals be used to perform other tasks (e.g. for AlfWorld only constructing a manual out of Put but testing on Put Two) - Despite the case conditioned prompting to mitigate hallucinations for the Builder, the Planner can ignore these rules as noted in the limitations. Is this the most common failure case and, if so, what were the practical solutions that were taken to make the Planner pay attention to the Builder's rules? - As a follow-up, AlfWorld and MiniWoB++ are environments where there is likely overlap in the LLM's training data and the benchmarks (home environment and web-based environments); however, how would AutoManual perform in environments that may not align with the LLMs priors (e.g, Atari games or Sokoban)

Rating

8

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations section is transparent and the broader impacts section addresses societal impact.

Summary

The authors propose a new approach to interactive environment interactions by LLMs which they dub AutoManual. The approach takes an LLM and uses it in several different roles, differentiated by what context is included in their prompts, and some number of saved rules. The fundamental idea of the approach is to optimize over the saved rules in order to improve the performance. The authors demonstrate that AutoManual can perform well with a single human-example in both ALFWorld and MiniWob++ compared to baseline approaches and they demonstrate the importance of its various components through an ablation study. Edit: I have updated my review based on the authors' response.

Strengths

The paper proposes what appears to be a useful framework for applying LLMs to decision making tasks. Clear arguments are made for the various components of the framework, which are all fully described in the appendices. The ablation study meaningfully demonstrates the additive value of the various components, and the comparison to baselines demonstrates strong performance.

Weaknesses

I have one concern with the paper and a few minor concerns. My major concern has to do with the reported performance of AdaPlanner. The authors acknowledge that they could not replicate the performance but justified this that AdaPlanner's format requirements were too strict. The original authors, Sun et al. identified that the issue with 3.5-turbo (and one would assume later models) was their tendency to hallucinate. If AutoManual is robust to these hallucinations that would be excellent, but it would make sense to include an AdaPlanner implementation with gpt-3 given that is the stated preferred models by AdaPlanner's authors. Further, the results with gpt-3 appear to meet or exceed the performance of AutoManual. This is by far my largest concern with the current paper draft and I'd strongly encourage the authors to include an AdaPlanner gpt-3 baseline or explain their reasoning not to. In terms of minor concerns, there are a number of claims that are somewhat overblown in the current paper text such as "to enhance their readability and global understanding" when no results demonstrate improved readability or global understanding. Similarly, the authors claim the approach is "user-friendly" and "like a teacher" in the same paragraph, given that the authors do not include any human subject study or other user tests these are unsupported in the current draft. "making it grounded and well-understood" would be yet another unsupported claim. Another minor concern is that some implementation details and associated design decisions are not given, such as not specifying the value of Nmax used for any of the experiments. Finally, the language is somewhat off in certain parts such as "the function should be called"->"this function should be called", "like a teacher conveys"->"like how a teacher conveys", and so on.

Questions

1. Why didn't the authors include a gpt-3 baseline for AdaPlanner? 2. Do the authors feel AutoManual handles hallucinations better? If so, why? 3. Can the authors given the value of N_max and other hyperparameters missing from the paper?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes the authors have adequately addressed the limitations.

Summary

The authors propose AutoManual, a method for improving LLM agents by collecting and updating rules online via rule-assisted interactions in the environment, where code form planning is used for the interactions. In addition to the management of rules, AutoManual also maintains skill and reflection libraries per task type, where successful trajectories form the skill libraries and failed trajectories form the reflection libraries. The authors empirically show that AutoManual outperforms the compared baselines in ALFWorld and MiniWoB++ environments. **Post-Rebuttal/Discussion**: While I appreciate the detailed author response, I don't think it fully addresses the major concerns I had. For example, by "maintaining the skill and reflection libraries per task type (L166)", I meant there exist explicitly and discretely isolated skill and reflection libraries, one for each task type. This design is made by the humans, not the LLMs. It may be doable for ALFWorld, which is comprised of clearly separable task types, but it would require more advancement for more complex environments. Therefore, I maintain my original score.

Strengths

- The proposed method, AutoManual, alternates between exploiting and updating the set of rules, which is sound as the rules also get updated online based on the relevant experience. - The authors show that the proposed method exhibits strong performance in ALFWorld and MiniWoB++, by comparing it empirically with multiple baselines. They also provide a fair amount of analyses including the ablation studies, which help to understand the proposed method in empirical aspects.

Weaknesses

- While the presented empirical evaluations mainly compare the number of examples used as a measure of the amount of domain or human knowledge provided to the methods, such knowledge can be injected in different forms, e.g., input prompts or the exposure of LLMs such as GPT to the benchmarks and corresponding execution codes during their training. But another point I notice is that, this work injects such knowledge in a more explicit or discrete form by maintaining the skill and reflection libraries per task type (L166), which contributes most to the performance according to Table 3. - Over-claiming statements need to be clarified or toned down. For instance, the Abstract claims that LLM agents "typically require elaborate design and expert prompts" in specific domains and suggests AutoManual as a solution. However, it appears that AutoManual leverages human-designed prompts for each domain as well as domain knowledge for design (as mentioned above). - The experimental settings for the GPT-3.5-turbo results may not be entirely fair. According to Table 1 from Appendix D, the "GPT-3.5-turbo" runs with all the baselines are actually based on GPT-3.5-turbo except for the Builder component for ExpeL. It means that the Actor and Self-reflection for offline trajectories for ExpeL are also equipped with GPT-3.5-turbo, unlike Planner and Builder for the building stage for AutoManual. - The manuscript can be improved in terms of presentation. especially the choice of technical vs non-technical terms. For instance, "free-form code" appears multiple times, but I couldn't find clarifications on what exactly "*free-form* code" is and whether (and how) it is different from just "code", which I assume is not. I have a similar concern about "natural programming capabilities" (L141).

Questions

* Are the "initial rules" made or selected by humans or fully automatically derived by Builder? * Please take a look at the Weaknesses section above.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors discussed the fair limitations in the Limitations section.

Re: Rebuttal by Authors

Thanks to the authors' for their detailed response! My major concern around Adaplanner has been addressed. I would suggest including a truncated version of that in the next version of the paper. As such, I have increased my score. I disagree with the characterization that the claims around human judgement are intuitive. I also do not agree that reviewers' subjective opinions would be sufficient to support these claims from the provided examples. I believe that arguments around human response to any system or system output require a human subject study for support.

Response to Authors

Thank you for the response. I appreciate running the temperature experiment - I am happy to see that the results are relatively robust and am excited to see this applied to other domains in the future! I have increased my score.

Reviewer Engagement

Dear Reviewer, Please remember to engage with the authors. Thanks! AC