VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

Summary

The paper proposes a new evaluation dataset called WebBench that tries to cover the issues of existing benchmarks being: failing to capture the unique features of web pages and to measure fine grained capabilities such as OCR, understanding and grounding. WebBench consists of 1.5k instances covering seven tasks: captioning, webpage QA, heading OCR, element OCR, element grounding, action prediction and action grounding. The authors evaluate 14 open-source MLLMs covering closed and open source models. The benchmark is constructed using different processes depending on the task. Some use LLMs and human verification, others are completely carried out by human annotators.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The benchmark covers a sufficiently wide range of tasks on website screenshots and it is definitely a useful resource for evaluating multimodal language models. The evaluation includes the main closed and open source LLMs.

Reasons to reject

There is one issue in the way the evaluation data is produced for one task, namely captioning. In this task, metadata that is mostly a sequence of keywords is converted into a more natural description with GPT-4V. In addition, the latter is seeing both screenshot and metadata. At evaluation time, each model is asked to generate the description. This whole process gives an unfair advantage to GPT-4V, which is exacerbated by using ROUGE-L as the evaluation metric for open-ended tasks. It is fair to say that gold and predicted descriptions would exhibit similar structure and wording. This raises a question about how much GPT-4V relies on the keywords to output the description, and how much information is actually coming from the screenshot. Maybe the model relies more on the image? On a similar note, the appendix shows that action descriptions are also suggested to annotators by GPT-4V. What if the model is generating and biasing the annotators towards actions that it can successfully perform? Would this possibility conflate GPT-4V performance on this task too?

Questions to authors

Is it correct saying that the number of unique screenshots is 139? Why is a discussion / related work on methods that relies on the DOM/source code to carry out tasks on web pages not in the paper? E.g. DocLLM: A layout-aware generative language model for multimodal document understanding For the heading OCR task, how often the H1 header is the right one? Why is Gemini Pro missing from Figure 4 and 6? What about using model-based metrics or critics for evaluating the open ended generation tasks?

Summary

The paper proposes WebBench, a fine-grained benchmark for webpage understanding. Specifically, the paper argues that existing web benchmarks for multimodal LLMs are focused on end-to-end or agent capabilities, conflating multiple challenges in one task and thus hard to pin down what is truly challenging for a tested model. WebBench has 1.5k samples from 39 websites. It breaks down web understanding into 3 levels: (1) website (2) element and (3) action. Under each level, there are elementary skills needed such as HeadOCR and Captioning for (1) and grounding for (2). The authors evaluated both closed-source and open-source multimodal LLMs on WebBench and found even the SOTA model (GPT-4V) achieves only ~65%, indicating room for improvement.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

+ Overall I find WebBench as a worthwhile effort to break down challenges in web understanding. Web understanding is known to be complex and often requires understanding of different levels and sometimes agentic workflow. It makes a lot of sense to test these capabilities individually instead of having one end-to-end task which can end up being too challenging and uninterpretable. + The authors tested a wide range of models, especially open-sourced ones. One nit-picking suggestion is to include more state-of-the-art closed source models such as Gemini 1.5 Pro and Claude 3 Opus.

Reasons to reject

I do not have a major reason to reject.

Summary

This work presents WebBench, a new webpage understanding benchmark of just over 1.5k samples. The benchmark is designed to target seven different web page related tasks: captioning, webQA, heading OCR, element OCR, element grounding, action prediction and action grounding. Using the new dataset, 18 different model variants (of 16 unique model architectures) are evaluated. Along with the quantitative task evals, there is substantial analysis, including correlation analysis among the tasks themselves, as well as the correlation between other vision-language scores with WebBench aggregate scores and the relationship between WebBench scores and image resolution size. The work also includes a couple of qualitative examples of failure cases and ablation studies for grounding. This paper presents high quality work with an appropriate amount of analysis and discussion. Its thorough benchmarking across many model variants is also appreciated. While the benchmark is not very large, this topic is a growing research area and having additional evaluation resources would positively impact the community.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

1. The paper proposes a well designed new benchmark for multimodal webpage understanding, consisting of a range of 7 different tasks. The tasks capture different levels of webpage understanding: the full webpage, element, and action level. 2. Along with the data there are substantial evals run across many models, providing comprehensive baselines, most of which are open source and easily reproduced. 3. There are numerous analyses and ablations provided, which verifying desirable qualities in the dataset and its utility in future work.

Reasons to reject

1. While more resources are certainly useful in this topic area, there are captioning and grounding datasets that would capture several of the tasks proposed in WebBench; either from prior human annotation efforts or automatically synthesized datasets. From the related work discussion and examples, it is not clear to me what distinguishes the dataset from prior work. [1] Yao et al. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS 2022. [2] Burns et al. A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. EMNLP 2023. [3] Srinivasan et al. WIT: Wikipedia-Based Image Text Dataset for Multimodal Multilingual Machine Learning. ACM SIGIR 2021. 2. Noise in task annotations. In the construction of the element grounding task, it's highly probable that there is noise in the annotations. For example, are there elements selected as negatives that could be congruent with the target element in action prediction task? 3. Another drawback is that it seems the tasks are annotated on separate samples, and ultimately there aren’t that many samples per task. This makes the benchmark a bit less well motivated because there are other datasets that cover the same task, and it would have been nice to have annotations for many tasks on a single screen, instead of them being non-overlapping / disjoint.

Questions to authors

Questions 1. How is the correlation between tasks in Figure 5 calculated exactly? Below are comments/suggestions for the authors 1. As mentioned in the reasons to reject, there are a couple of highly relevant related work that I would suggest adding to the prior work/citations: [1] Yao et al. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS 2022. [2] Burns et al. A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. EMNLP 2023. [3] Srinivasan et al. WIT: Wikipedia-Based Image Text Dataset for Multimodal Multilingual Machine Learning. ACM SIGIR 2021. 2. The last sentence in Figure 3 seems unnecessary, it was probably from an earlier version of the draft (I'd suggest removing). 3. Figure 2 is never referenced in text.

Summary

This work presents “WebBench,” a new benchmark for evaluating multimodal language models (e.g., image, text → text), specifically focusing on applications involving reasoning over webpages. The benchmark consists of ~1.5K example prompts (webpage/input/response tuple) spanning 139 individual websites, and **7 task families** each with their own evaluation metric: 1. *Captioning* (ROUGE Score): Given a webpage screenshot, generate a natural language summary of the website (where ground-truth is a GPT-4V rewrite of the HTML “meta” tag) 2. *WebQA* (F1 Score): Given a question about the webpage (written by a human annotator) generate the answer (open-set generation). 3. *Heading OCR* (ROUGE Score): Generate the heading text for the given webpage (extracted from the first <h1> tag). 4. *Element OCR* (ROUGE Score): Given a webpage with a highlighted DOM element > 20 words (drawn bounding box), transcribe the given element in natural language. 5. *Element Grounding* (Multiple Choice Accuracy): Given a human-provided description of an HTML DOM element, and 8 potential candidates (drawn bounding boxes), select the candidate element that best matches the description. 6. *Action Prediction* (Multiple Choice Accuracy): Given a highlighted DOM element (with an href tag), select the title of the “target” webpage given a set of candidates (with negatives scraped from other elements of the same webpage). 7. *Action Grounding* (Multiple Choice Accuracy): Given a human-provided high-level instruction (e.g., “search for flights to Montreal”) and eight candidate DOM elements (drawn bounding boxes), identify the candidate element that is most relevant to the instruction (e.g., search bar for “Flights”). Beyond curating this extensive benchmark, the work also evaluates and performs a thorough analysis of 14 state-of-the-art multimodal language models (14 open-source, 2 closed-source) across all tasks, including a stratified breakdown of how models behave as a function of scale (7B vs. 13B vs. 34B), input resolution, and manual error analysis. The appendix provides additional detail as to the data annotation and verification process. The full evaluation benchmark and evaluation harnesses for open-source models will be released.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

This is a strong paper introducing a new evaluation that 1) fulfills a demonstrated need given the recent proliferation of multimodal language models, and 2) clearly distinguishes itself from prior web-based evaluation benchmarks such as Web Shop and Visual WebArena by focusing on clearly defined and fine-grained “subtasks” necessary for reasoning over websites. While the overall sample size per each of the 7 subtasks is on the smaller side, I do believe the existing examples are clean and high-signal. The effort the work makes to leverage human annotators to both provide input prompts and verify outputs is clear, and further strengthens the benchmark. Finally, the analysis of existing models demonstrates that this benchmark is not only valid, but that there are subtasks with a ton of “headroom” — showing that this evaluation will be useful to the field and those working on developing stronger MLLMs in the future.

Reasons to reject

There are a few minor issues that I believe the authors can easily fix for the final submission. First, it’s a bit strange to me that the evaluation metric for all of the OCR-based subtasks (e.g., Element OCR, Heading OCR) is ROUGE score rather than exact match (or F1 score); if we want high-fidelity OCR, shouldn’t we be penalizing our MLLMs for any generations that deviate from the ground-truth? Furthermore, some of the details around the annotation procedure are missing (how annotators are selected, which annotators are used for generating which data per subtask, which data the authors of the work actually verified vs. external annotators). It would be really great to be specific and clear about these details in the final version of the paper.

Questions to authors

Were external annotators only used for the action grounding task? If not, could you provide a description of the annotation pipeline used for the other subtasks (similar to Appendix A)? Beyond the webpage screenshots and input prompts provided in the benchmark, can you clarify what other metadata about the webpages you plan to release? Minimally, it would be great to get original URLs, access dates, as well as code for rendering websites (e.g., the Playwright pipeline described in the paper).

Post-Rebuttal Response

Thank you for the explanation of the ROUGE-L vs. F1 metrics; I think this is an important discussion, and I'm glad to see these results. I'm also very excited that data and rendering code will be open-sourced! This was a very strong paper to begin with, and my opinion hasn't changed; I definitely will be pushing for acceptance at this time!