SteP: Stacked LLM Policies for Web Actions

Summary

They advance the research on the line of performing web tasks with LLMs by decomposing them into different policies (instead of using a single prompt). In particular, they develop an approach based on a Markov Decision Process where the state is compounded by a stack of policies, which represent the sequence of policy calls. Evaluation of their methodology across key benchmarks such as WebArena, MiniWoB++, and a CRM simulator reveals promising results, demonstrating efficiency in data consumption.

Rating

8

Confidence

2

Ethics flag

1

Reasons to accept

- The authors introduce an innovative method for addressing the challenge of executing web tasks with LLMs. - They conduct a comprehensive comparison with other baseline methods across major benchmarks. - Their approach demonstrates superior performance over the state-of-the-art in certain benchmarks, all while necessitating less data. - The paper is well structured and presents information in a clear and accessible manner. - Following acceptance, the code will be made publicly accessible. - I believe it encompasses a satisfactory amount of work and quality to merit acceptance at COLM.

Reasons to reject

- The absence of a Conclusion section, which consolidates the conclusions drawn from preceding sections, is noted. However, aside from this observation, I do not identify any other issues with this paper.

Summary

This paper proposed a method called Step, which aims to solve the complex web navigation tasks. Instead of relying on one global policy/prompt to control how agent interact with the environment, Step contains a set of policies that are tailored to solve different sub-tasks, and the Step agent dynamically composes/invokes different policies to solve the task at hand. In particular, each policy can choose to execute an action in the environment, call other policies or call itself, which offers great flexibility in agent’s exploration. The Step method is evaluated on popular web agent benchmarks including WebArena, MiniWob++ and CRM and a few real-world websites. The results show that it outperforms previous methods and baselines with a flat policy.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The proposed method is very intuitive that the complex tasks can be decomposed into simpler ones, and each simple task can be handled by dedicated policies. The results on web agent benchmarks are also pretty promising. Overall the Step agent uses less tokens in its prompt to complete the task compare to previous baselines, hence that it is more cost effective.

Reasons to reject

My biggest concern is about the construction of policies. As the authors admitted in the limitations, the Step method requires user to manually write detailed prompts for every policy. This implies that the method cannot be easily scaled to more websites/tasks. Moreover, from the examples shown in Appendix B.2 we can see that these policies/prompts are very tailored towards specific websites. For example, search order policy informs the agent to first go to My Account and then go to My orders. I’m not sure if the prompt can transfer well out of the box to different websites. For example, what if a website do not have the My Account button but the user actually needs to click their profile picture to get to the account details? In section 5.1.2., the authors mention that the manully written policies coveres at least 50 intent templates from WebArena. Have you checked if the gains on WebArena are mainly from these covered intents? Missing references: The idea of constructing specific policies/prompts at different states to control the web agent has been explored in Ma et al. 23. It would be good to compare your method with their work as well. Ma, Kaixin, Hongming Zhang, Hongwei Wang, Xiaoman Pan and Dong Yu. “LASER: LLM Agent with State-Space Exploration for Web Navigation.” ArXiv abs/2309.08172 (2023): n. pag.

Questions to authors

Please see the limitation section

Summary

This paper proposes SteP, which dynamically composes policies from a policy library into a policy stack, to solve web tasks. The proposed model shows strong performance on WebArena, MiniWoB, and a CRM task.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- The stack formulation for composing policies is intuitive and effective. - The performance is strong on multiple tasks. - The paper is clear and well written.

Reasons to reject

My main concerns are as follows: - The task-specific nature of the proposed method, as it relies on a manually crafted choice of policy library, as well as demonstrations for the policies. - A lack of novelty with respect to existing decompositional methods such as [1,2], as both methods explored the utilization of dynamic decomposition using a predefined skill library. [1] Decomposed Prompting: A Modular Approach for Solving Complex Tasks. Khot et al., ICLR 2023 [2] ADaPT: As-Needed Decomposition and Planning with Language Models. Prasad et al., NAACL 2024

Questions to authors

- What kind of prompts does the meta-policy need for each task, in order to manage the policy stack? - Can the authors share more details about how the policy library was chosen on each task?

Response to authors

I appreciate the authors' response. I understand the distinction from DecomP and ADaPT better now. As I understand it, it offers a much more flexible control of policy composition than those methods. That said, in order to be complete, I think paper needs to include the full policy library for all tasks, since these are all handcrafted for each task. Overall, I think the method is interesting and I have increased my score accordingly.