Summary
This paper investigates the evaluation of large vision-language models (LVLMs) and the currently used benchmarks. Within the paper two primary issues are identified: the lack of need for visual information, and data leakage. Based on these issues a new compiled benchmark is proposed MMStar that includes a set of multi-modal samples validated by humans, and a variety of models are evaluated on this new benchmark.
Strengths
With the rapid progress of the field of LLM and LVLMs, it is crucial that we have reliable benchmarks for evaluation - this work explores existing benchmarks, identifies potential issues therein, and proposes a new benchmark which contributes to the quality of evaluation in the field. Additionally, the work proposes new metrics for evaluation and performs a benchmark evaluation of various models.
Weaknesses
1) Random Chance
An issue in this work is that it insufficiently accounts for random guessing and data bias. As shown in [27], the ground-truth chance for it being answer A or B on MMbench are 26.4% (due to having questions with less than 4 options), and moreover, certain models may prefer certain answers. LLMs may get correct answers even when they require visual input. Given the 26.4% baseline for MMbench from [27], it is also surprising that the random choice value reported in this paper, for MMbench, is 0.0 in both Table 1 and Table 2. Moreover, an additional baseline based on majority class may also be necessary here.
For the results in Figure 2, which require 6 out of 8 models to get a hit it is unlikely that random guessing has a big influence. However, for the results in Table 1 this is not the case. For instance, it appears that all MMBench results in Table 1 are below the 26.4% from [27] - which means all LLM do worse than always answering A (or B). For other benchmarks there may similarly be data biases in which answer is more frequent.
Relatedly, it may be the case that such data biases about which options are more frequently chosen are more pronounced for multi-modal questions, i.e., when asked about colour the answer is more often grey across all datasets and settings - which means LLM do not learn this data bias as their training data does not include such questions, but LVLM may learn this because the same bias is present in their training data. While this is unlikely to fully explain the phenomenon observed in Table 2, it may explain part of it and not be directly related to data leakage.
2) Not all benchmarks
The issues identified with existing benchmarks do not hold evenly across the benchmarks tested. In particular, MMBench and slightly lesser, MathVista seem to do pretty well with respect to these issues. This also appears to be reflected in the construction of MMStar, where after Manual Review the proportion of questions from these two existing benchmarks jumps considerable - in the end making up almost half of the MMStar benchmark. Which raises questions whether there is any benefit of using MMStar versus simply using MathVista and MMBench.
3) Manual review
The manual review description is insufficiently clear. From the three criteria applied, only the first one is somewhat clear - for the other two it is unclear how the 'experts' judged this. I would expect description (can be in appendix) of agreement rates between these experts for these criteria, as well as further description of what these criteria entail.
4) New metrics
The newly proposed metrics MG and ML are somewhat unclear in what they measure. If the 1500 questions in MMStar all require visual input as determined by the manual review, then what additional information does the MG metric give? Given the discussion above about chance, it appears the MG metric is more of a correction for random guessing. The ML metric similarly doesn't account for random guessing, or the aforementioned potential for data biases.
Questions
The paper raises a number of interesting questions and points out limitations of (some) existing benchmarks. Unfortunately, the data leakage issue appears to be not fully disentangled from the potential of data biases. I would appreciate input from the authors on this point.
Additionally, I would be interested in discussion on random guessing within the paper, and whether the ideas presented in [27] may address this - and then subsequently how this influences the findings in the paper. Also taking into account the majority class versus random guess.
Limitations
I believe question 9 in the paper checklist has been answered incorrectly - or at least the justification given is not valid. Given that the paper discusses a benchmark that may include images of people or copyrighted material it is crucial that the authors affirm whether the work has been done in accordance to the ethical guidelines. Even if the data is a compilation of existing datasets - by selecting and combining information from these the new resulting dataset may be biased in ways the original datasets were not (e.g., by selecting only those images containing users from a certain demographic).