Language Models can Solve Computer Tasks

Summary

The paper introduces a novel approach, Recursively Criticizes and Improves (RCI), for guiding large language models (LLMs) to execute computer tasks using natural language commands. The RCI method outperforms existing methods and sets a new state-of-the-art on the MiniWoB++ benchmark when combined with the InstructGPT-3+RLHF LLM. The authors also demonstrate that RCI enhances LLMs' reasoning abilities on natural language reasoning tasks, and performs better when combined with the chain of thought (CoT) prompting.

Strengths

1. The paper is overall well-written and easy to follow. 2. The paper introduces a new method, Recursively Criticizes and Improves (RCI), which guides large language models (LLMs) to execute computer tasks using natural language commands. The idea of explicit and implicit RCI is interesting. 3. Abundant experiments have shown that RCI not only shows great reasoning ability on different tasks, but also achieves same performance with fully-supervised methods on real-world computer tasks.

Weaknesses

There are a lot of works that leverage environmental feedback or model self-reflection to improve the plans, like self-debugging, reflexion, self-refine, etc. The authors have briefly introduced some of them in the appendix, but a detailed comparison of these methods is still needed. The baselines included in the reasoning part of experiments are not enough as well.

Questions

I am wondering how RCI performs compared with other LLMs-based methods that leverages environment feedback and self-evaluation.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors include the limitations in the model failure part.

Summary

The paper presents a new method to enable LLMs to execute computer tasks guided by natural language. The method uses a simple prompting scheme called Recursive Criticism and Improvement (RCI), which prompts the LLMs to find and fix problems in their output. Without large amounts of expert demonstrations and task-specific reward functions, RCI improves the performance of LLMs on the MiniWoB++ benchmark, a web-based simulation environment that offers a diverse range of computer tasks. The paper also demonstrates that RCI enhances the reasoning abilities of LLMs on a suite of natural language reasoning tasks. The paper claims that RCI is a significant contribution to the development of intelligent agents that can solve any computer task by communicating via natural language.

Strengths

1. This paper is original and insightful in tackling the grounding problem of LLMs by introducing RCI which uses the self-criticism and self-improvement abilities of LLMs to finish computer tasks and reasoning tasks. 2. It seems that RCI is general and flexible enough to be integrated with other techniques like CoT. 3. The paper applies RCI to a diverse range of web-based computer tasks and natural language reasoning tasks and shows promising results. 4. The paper is well-written and clear in presenting the main ideas, contributions, and results.

Weaknesses

1. The impact of the RCI iteration steps should be presented. Will RCI iteration always increase the performance? 2. There is no explicit criterion for terminating RCI, and it relies on multiple attempts on the dataset. However, this kind of trial may not be feasible for new tasks in the real world. A simple approach might be to iterate as much as possible, but this would increase computational and time costs significantly. 3. Although RCI performs well in various reasoning tasks, considering the wide variety of models available now, it is recommended that the author conduct tests on several different models to demonstrate the effectiveness of RCI. For example, zero-shot CoT [1] has measured 17 different models in total. Another example is that few-hot CoT [2] has also measured models with different sizes (see their Table 4). Also, newly proposed models like Vicuna should be measured if possible. [1] Large Language Models are Zero-Shot Reasoners [2] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Questions

1. The model name “InstructGPT-3 + RLHF” in the experiment part of the main paper is quite confusing since there are 3 different models named after it in the appendix. Also, InstructGPT is also tuned with RLHF. I suggest replacing them with corresponding API names. 2. Why are benchmark AQuA, CommonSenseQA, and StrategyQA evaluated on the zero-shot setting but not evaluated on the few-shot setting? 3.“Specifically, we employ three models, namely, GPT-3 (davinci), InstructGPT-3 (text-davinci-002) , and InstructGPT-3 + RLHF (text-davinci-003, gpt-3.5-turbo, gpt-4).” Which InstructGPT-3 + RLHF model is used in section 3.2? 4. Line 246 – 252: “Our findings reveal that our agent surpasses SotA in supervised learning by 34% and in reinforcement learning by 24%. Notably, our proposed RCI prompting method outperforms the SotA LLM approach, even when the latter employs both finetuning and few-shot examples in prompts. This outcome highlights the effectiveness of our approach in extracting vital knowledge for computer tasks from language models. However, our agent underperforms in comparison to SotA in the combination of supervised and reinforcement learning, which employs significantly more expert data and online interactions.” This result is inconsistent with Figure 4 (a). I guess there is a typo in Figure 4 (a) where “Ours” should be 0.91 instead of 0.94.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have not discussed their limitations.

Summary

This paper proposes a prompting approach that utilizes a critic to improve the web navigation performance. Within the same context, the model reviews a candidate answer and improves it based on reasoning steps. For web navigation, there are two phases: explicit and implicit RCI. Explicit RCI takes a task and generates a plan with step-by-step actions. This plan is fed to the implicit RCI to ground on the state and the agent. Resulting action is executed in the environment to retrieve a new state. Compared to previous SL and RL approach, RCI achieves competitive to the SoTA while using only few-shot prompting.

Strengths

The paper introduces a novel prompting approach with planning and iterative improvement. It performs favorably to CoT prompting and outperforms previous finetuning approaches while using significantly less data. It also shows the general capabilities of LLMs in decision making tasks and illustrates limits of the MiniWoB benchmark.

Weaknesses

Some questions regarding better baselines and RLHF performance. - More detailed comparison with recent approaches such as ReACT or InnerMonologue is missing. What is the main novelty of RCI and why would it be a better candidate for computer tasks compared to ReACT which also studies WebShop benchmark tasks. - Instruction tuning and RLHF are crucial for achieving good results, otherwise the success rate drops to 50%. Given that training data for RLHF training is undisclosed, I wonder if similar tasks (or even MiniWoB data) are used during training. Is there any empirical evidence for the case of InstructGPT-3+RLHF model? - WebN-T5 doesn't use in-context examples. It uses the history of actions within the same episode. (referring to Figure-4 in your paper) - WebGUM and WebN-T5 also don't use dictionary based typing actions. WebGUM is missing from comparison. - Please clarify Line 33 as there are previous models using LLMs that you also compare.

Questions

1. How do you compare to more recent reasoning approaches? 2. Could RLHF training be using some web navigation tasks or even MiniWoB?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses some of the limitations but more detailed discussion on generalization to real websites would be helpful.

Summary

This paper proposes an algorithm for executing tasks using LLMs. While this can in principle be applied to any sequential task with language input and output spaces, they tested primarily on computer control tasks. The method, RCI, works as follows: - The environment state, action space, and task are represented in language (e.g. HTML code for the state). - Based on the task description, the LLM generates a high-level plan of how to solve the task. - The LLM writes a critique of its plan and improves it. This can happen multiple times. - Conditioned on the plan and the current state, the LLM is asked to find the webpage elements necessary to execute the plan. - Based on the above, ask the LLM to generate an executable action. They show that on a large subset of Miniwob++ tasks, this method is able to outperform past LLM SOTA and is able to match past IL+RL SOTA without needing any training. They also show improvements over other prompting methods on various math and reasoning tasks.

Strengths

- The method gives a substantial performance boost compared to all Miniwob++ baselines (except one which uses far more demonstration data). It also shows consistent improvements in math and reasoning tasks. This is useful since it suggests that for many computer control tasks collecting lots of demo or RL data is unnecessary. - Each component of the pipeline includes intuition for why it's necessary + ablations showing that performance drops significantly without it. - It's hard to judge novelty given the recent explosion in works prompting LLMs for tasks with various forms of self-reflection. I'm inclined to say it's all concurrent work.

Weaknesses

- See the "Questions" section for discussion on figure/discussion mismatch. - This method has limitations, including (a) it is hard to solve tasks where the HTML description exceeds the context width and (b) the agent is currently unable to handle tasks which involve dragging or other complex UI manipulation. Currently, these tasks are excluded from the set of evaluation tasks. On the other hand, tasks where the prior SotA did poorly were stated to be intentionally included. This seems like a bit of an unfair comparison. (Though I think many of these problems could be addressed by modifying the action space and getting a longer-context LM). Even if the additional tasks are not reported in the main results I think it should at least be more prominently emphasized that tasks where RCI has insufficient context were modified or excluded.

Questions

**Major questions** * The description of Figure 4a doesn't seem to quite match the numbers in the table. "our agent surpasses SotA in supervised learning by 34% and in reinforcement learning by 24%". (I'm guessing you meant 37 and 27. Also, please be clear whether you're talking about *percent improvement* or *percentage point improvement.*). Also, the text says "our agent underperforms in comparison to SotA in the combination of supervised and reinforcement learning..." but the graph currently shows they match. Which is correct? **More minor** * Nit: Typo L196 "wich" * Nit: L322 "exist" -> "exists" * Explain more clearly what "dictionary-based typing actions" are. * In Figure 4a, it would be helpful if the caption stated which models are the "SL SotA", "RL SotA", and "SL + RL SotA". Currently it isn't clear.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are discussed, although additional limitations (such as lack of history, limited context window, and inability to improve through accumulated agent experience) are not discussed much.

Thanks the Authors' Responses

Thanks the authors for their responses. I will raise my score to 6, as most of my concerns have been resolved.

Thanks for your response!

I think solving computer tasks using a strong prompting approach is valuable, so I will maintain my score. But, I am still skeptical of the proposed approach's generalizability to other models other than InstructGPT-3+RLHF. Related to that, the question of whether any web navigation related task is utilized in RLHF or instruction tuning makes the contribution of the proposed approach less clear.