Thank you for your insightful feedback! We’re glad to hear that you found the work significant and that it effectively addresses a critical gap in the field of AI web agents through the use of APIs.
We would like to address the following concerns:
> How does the hybrid agent decide when to switch between API calls and GUI-based interactions? The paper does not provide a detailed explanation of how the hybrid agent decides when to switch between API calls and GUI-based interactions.
Thank you for pointing this out! We added a more detailed explanation on how the hybrid agent works in Section 4, and we provided example prompts to the hybrid agent in the appendix. Specifically, the hybrid agent is provided with both a browsing action space and access to APIs for each website. We use simple prompts and offline the decision-making between browsing and API entirely to the agent. Empirically, the agent usually chooses API whenever it is available, and switch to GUI browsing in other cases. We also want to highlight that this design is effective, achieving 35.8% task success rate on WebArena, outperforming many existing approaches with heavy human efforts.
> Expanding the evaluation to include a wider variety of websites with different levels of API support and varying complexities would provide stronger evidence of the agent's effectiveness and robustness
We have added more discussion on why we chose WebArena as the evaluation benchmark in the revised paper. Specifically, in Section 2.1 and Limitations, we discussed existing benchmarks such as MiniWob, Mind2Web, WebVoyager, and VisualWebArena, and we discussed why we chose WebArena over the other benchmarks. We have also softened our claim to "Other benchmarks, such as Webshop (Yao et al., 2022), MiniWoB (Shi et al., 2017), Mind2Web (Deng et al., 2023), WebVoyager (He et al., 2024b), and VisualWebArena (Koh et al., 2024a), provide alternative evaluation platforms. However, as discussed in Section 2.1, WebArena aligns closely with real-world scenarios and our use case, while other benchmarks lack support for API calling."
> How does your agent manage situations with incomplete, undocumented, or frequently changing APIs? The paper does not address how the agent handles incomplete, undocumented, or changing APIs.
Our work acknowledges and addresses the limitation of API only agents – We propose hybrid agent, as extensively introduced in Section 4. The corresponding experiments are in Section 5. In such cases, the agent could use web browsing to handle tasks with incomplete, undocumented, or changing APIs, and this hybrid agent achieves the best performance compare to browsing-only and API-only agents.
> Direct interaction with web service APIs raises potential security and privacy concerns, such as authentication management, rate limiting, and compliance with terms of service.
Thank you for pointing this out! We acknowledge the potential risks. We would like to kindly note that addressing the security and privacy concerns are orthogonal to our work.
> The evaluation primarily reports success rates without sufficient analysis of other important performance metrics such as execution time, resource utilization, or learning efficiency.
In Section 6.1, we discussed the average steps and costs the agents take on WebArena tasks. We believe that the average steps serve as a measurement for execution time while costs serves as a measurement for resource utilized. For example, for the shopping admin website, the average cost of the hybrid agent is $1.4, while the average number of steps is 9 steps.
> Websites frequently update their interfaces and APIs, which can break automation scripts. The paper does not discuss how the agent adapts to such changes over time. Exploring methods for the agent to detect and adjust to updates would improve its long-term usefulness.
Thank you for your suggestion! We agree that interface and API update of the websites could bring potential challenges. In our case, we can always provide the *latest* API docs to assist the agent in understanding the new information.