Multimodal Web Navigation with Instruction-Finetuned Foundation Models

Author Response to Reviewer idei

We thank the reviewer for the careful reading and constructive feedback. We address your concerns below and please let us know if you have further questions. **> Weaknesses 1** The major difference between the recent multimodal LLMs and our WebGUM is the task they can deal with. The recent multimodal LLMs, such as LLaVA and InstructBLIP, focus on static inference tasks like visual reasoning or Science QA. In contrast, this paper tackles web navigation, an interactive decision making problem, which requires temporal understanding and planning over the multiple timesteps. Decision making for autonomy and Static VQA are orthogonal research fields that have their own specific challenges. We have clearly explained the focus of this paper in the abstract and introduction. **> Weaknesses 2** We have investigated the architectural choices for base language models, comparing Flan-T5 (220M, 770M, 3B, 11B) and Flan-PaLM-8B. Flan-T5 is the encoder-decoder model and Flan-PaLM is the decoder-only model. In Figure 4 (right), while Flan-T5 shows a clear scaling trend, Flan-PaLM-8B performs sub-optimally (72.8%), compared to similar scale Flan-T5 models (770M: 72.4%, 3B: 75.5%, 11B: 79.0%). **Our results imply that in web navigation, encoder-decoder architecture (T5) is better at understanding HTML than the decoder-only model (PaLM, LLaMA, etc)**, since encoder-decoder architecture can capture the hierarchical and nested structure of HTML well. Our results also suggest that advanced language models are not always the best for downstream finetuning tasks. Because both Flan-T5 and Flan-PaLM are instruction-tuned on the same Flan dataset, our comparison should be more fair than the case with LLaMA. Moreover, while recent LLMs employ larger model sizes (LLaMA-7B + ViT-L14 for LLaVA, LLaMA-7B/13B + ViT-G14 for InstructBLIP), WebGUM only has 3 billion parameters (Flan-T5-XL + ViT-B16). Such parameter efficiency is a desirable property for autonomous agent research because it ensures on-premise deployment and low latency. Lastly, in the revised paper we have included the works you suggested (LLaVA, Otter, InstructBLIP) in the extended related work (Appendix B).

Author Response to Reviewer cMQF (1/2)

We thank the reviewer for the thoughtful review and comments. Please let us know any remaining questions or concerns if you have. **> Weaknesses 1** As described in Appendix C, our WebGUM accepts 4096 tokens as input, which might be the soft upper limit for dense attention models considering the computational costs. As the reviewer pointed out, in the evaluation of WebSRC (Appendix E), we have preprocessed HTML to shorten it, because in some train/test sets, HTML has over 10K tokens. For web navigation tasks with much longer HTML, we could incorporate (1) ranking language models [1] to prioritize HTML elements, (2) extractive summarization of HTML [2], or (3) long-context pre-trained language models with sparse attention [3], in the pipeline. In fact, we used the cached ranking results, released by the author, for Mind2Web, and we demonstrated the applicability of such a pipelined approach by achieving the best performance among the baselines. **> Weaknesses 2** Thank you for pointing this out. We updated Figure 1 and 5 with high-resolution images in the revised paper.

Author Response to Reviewer cMQF (2/2)

**> Questions 1** We have included a failure example of WebGUM in MiniWoB in Appendix L. In addition to that, we increased the number of examples in the revised paper. Please take a look at Figure 11 (Appendix L). For instance, WebGUM suffers from (1) ambiguous instruction, such as small items (`count-shape`), (2) confusion with the repetitive structure on the page (`social-media-all`), and (3) long-horizon (`guess-number`) tasks that may require memory. **> Questions 2** We have included the successful episodes on WebShop in Table 13. In the revised paper, we added the example response on Mind2Web in Table 14 (Appendix L). ``` [1] Deng et al., (2023) Mind2Web: Towards a Generalist Agent for the Web (https://arxiv.org/abs/2306.06070) [2] Gur et al., (2023) A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (https://arxiv.org/abs/2307.12856) [3] Guo et al., (2021) LongT5: Efficient Text-To-Text Transformer for Long Sequences (https://arxiv.org/abs/2112.07916) ```

Author Response to Reviewer a21F (1/2)

We sincerely appreciate your thorough examination of our paper and your insightful feedback. We have carefully considered your concerns and offer the following clarifications and additional information. Please let us know if there are remaining questions or unclear points. **> Weaknesses 1** We have proposed a simple yet effective recipe to build multimodal web navigation agents, leveraging the prior knowledge of the web environment from instruction-finetuned language models and visual encoders. We believe that our paper has the following important technical contributions: (1) We have introduced temporal and local visual tokens to perceive visual information better (Figure 2 and Section 5.1). Given the interactive nature of web navigation on computers, it is essential to consider both temporal and local relationships on the screen. Our experimental evaluation in Figure 4 (left) indicates that employing both temporal and local perception tokens is most critical for the performance, while the effects of various pre-trained ViT with different datasets or self-supervised objectives are marginal. (2) We have investigated the architectural choices for base language models, comparing Flan-T5 (220M, 770M, 3B, 11B) and Flan-PaLM-8B. Flan-T5 is the encoder-decoder model and Flan-PaLM is the decoder-only model. In Figure 4 (right), while Flan-T5 shows a clear scaling trend, Flan-PaLM-8B performs sub-optimally (72.8%), compared to similar scale Flan-T5 models (770M: 72.4%, 3B: 75.5%, 11B: 79.0%). Our results imply that in web navigation, encoder-decoder architecture is better at understanding HTML than the decoder-only model, since encoder-decoder architecture can capture the hierarchical and nested structure of HTML well. (3) We have shown a good estimation of the necessary amount of training dataset for multimodal web navigation. Previously, the community only had 12K, HTML-only dataset, and CC-Net implied that 2.4 million episodes and billions of frame online interactions would be needed to realize human-level performance. Our human-level performance with 401K demonstrations, and the newly introduced 347K multimodal dataset should encourage the community to develop capable on-premise multimodal agents via safe offline training. **> Weaknesses 2 & Questions 1** As described in Appendix F, we run LLM agents with 10,000 episodes per task and only keep successful trajectories to maintain the quality of the dataset. Such reward filtering is a common practice in web navigation as done in Humphreys et al., (2022) [2]. Because our dataset only consists of successful episodes, they do not have noisy data (i.e. failure episodes). We also added qualitative analysis in Table 15 (Appendix L) comparing the episodes from our 347K dataset and the previous 12K dataset. This shows that while previous work has included "hesitant" behaviors (e.g. clicking the same checkbox several times), our dataset has "shortest" behaviors. The quality of our dataset is appropriately controlled. **> Questions 2** "rollout" means "deploy" here. We deployed WebN-T5 (i.e. LLM policy) on MiniWoB environments, and WebN-T5 was asked to solve 100 episodes per task for all 56 tasks. The input and output are the same as our WebGUM; taking HTML, instruction, and action history as inputs, and predicting dictionary-format action (as described in Figure 2). **> Questions 3** Yes. Following Gur et al., (2022) [1] and Humphreys et al., (2022) [2], we evaluate WebGUM with 100 episodes per task. While other LLM baselines (RCI, AdaPlanner, Synapse) are evaluated with only 50 episodes, our evaluation with more episodes could reduce the variance during the evaluation.

Author Response to Reviewer a21F (2/2)

**> Questions 4** We’d like to note that, in web navigation literature, the number of parameters does not always correlate to the performance. For instance, CC-Net [2], trained through supervised learning and reinforcement learning, only has small-scale architectures compared to LLMs (transformer with 8 layers, 8 heads, and a 512-dimensional embedding, a dual-layer LSTM with 512 hidden units per layer, and 4 blocks ResNet). However, CC-Net can exhibit human-level web navigation performance (93.5%). Among the human-level web navigation methods (CC-Net, WebGUM, RCI/Synapse (GPT-3.5/4 with prompting)), there are trade-offs between model size, training cost/risk, and inference cost. Our method provides promising solutions among those by (1) mitigating training costs and risks through offline supervised finetuning and (2) achieving on-premise agents with low inference costs. | | CC-Net | WebGUM | RCI / Synapse | |--|--|--|--| | **Model Size** | small | up to 3B parameters | API-based private LLM | | **Training Cost/Risk** | High (RL) | Low (SL) | -- (in-context)| | **Inference Cost** | Low | Low | High | In addition, we have conducted an out-of-distribution evaluation in Section 5.3 (Figure 5, and 6), and a transfer learning evaluation in Section 5.5 (Table 3). WebGUM performs the best among competitive baselines. In particular, WebGUM achieved 78.5% success rate for unseen combinational tasks, outperforming Synapse (73.8%) a SoTA web navigation agent in Table 1. In real-world action prediction tasks from the Mind2Web dataset, WebGUM exhibits strong transfer achieving the best performance in all the metrics. These results support that WebGUM is generalizable enough to other web navigation tasks beyond overfitting. ``` [1] Gur et al., (2022) Understanding HTML with Large Language Models (https://arxiv.org/abs/2210.03945) [2] Humphreys et al., (2022) A data-driven approach for learning to control computers (https://arxiv.org/abs/2202.08137) ```