MileBench: Benchmarking MLLMs in Long Context

Summary

The paper presents a new benchmark dataset for multimodal (image+text) LLMs, called MileBench. The novelty of this benchmark is in its focus - testing multimodal models in long contexts with many images. The collected dataset is a subset of existing datasets and synthetically generated examples. The examples from the existing datasets were picked in a way which agrees with the benchmark's focus. The resulting dataset consists of around 6k examples categorized in 12 different tasks. The authors also measured the performance of various models on this benchmark and report the results. Notably, the closed-source models emerged as clear winners on most of the tasks. Overall, the paper is well written and presents a relevant and potentially useful contribution.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

* A new benchmark dataset which might be useful for multimodal LLM research. * Includes baseline scores for many publicly available models, making comparison using this benchmark more accessible.

Reasons to reject

* It is not very clear which datasets have been used for the data collection. References are given to the tasks, so I am assuming the data comes from there, but this should be listed explicitly, perhaps in the appendix. * Inadequate attention given to possible data contamination in discussion about the results (see my question below).

Questions to authors

* Have you tried using different metrics than ROUGE-L? I think for open ended generation tasks it might make more sense to use a LLM-based metric instead. I realize that selecting a good metric is secondary to this paper's focus, but by introducing a benchmark like this, you basically dictate which metric to use if someone else decides to use the benchmark to compare with other models. * Given that the collected benchmark is a subset of existing datasets, is it possible that the success of the closed-source models is determined by a data leakage? (https://aclanthology.org/2024.eacl-long.5/) Why are these models omitted in the analysis in section 4.3.3? Some minor suggestions: For better clarity, Table 2 should indicate which numbers are accuracy and which are Rouge-L scores.

Summary

This paper focuses on evaluating the long-context understanding capabilities of multimodal large language models. The authors introduce MILEBENCH, a comprehensive benchmark encompassing multiple dimensions of multimodal long-context understanding, such as Temporal Multi-image Understanding, Semantic Multi-image Understanding, Needle in a Haystack, and Image Retrieval. Additionally, the paper provides a brief evaluation of various multimodal models, ranging from closed-source to open-source, trained on images and videos, presenting a comparative analysis of their performance.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. Long-context capabilities, especially in the domain of multi-image, long-context understanding, are crucial functionalities of multimodal large language models, yet they are absent in many current models. A comprehensive benchmark can facilitate the development of these capabilities. 2. This paper provides an interesting and significant insight: numerous models that excel in common benchmarks show weak performance in multi-images scenario. 3. This paper is well organized, offering concise experiments across a range of popular multimodal large language models.

Reasons to reject

1. This paper collects data from publicly available datasets/benchmarks, yielding only 200 samples for each task. However, some of the pre-existing datasets used herein (e.g., START [1]) encompass a vast collection of video clips and annotations potentially suitable for directly assessing multi-image capabilities. It's not clear whether these datasets alone are inadequate. The authors can provide substantial evidence to justify the necessity of the proposed benchmarks. 2. Recent developments have seen certain models displaying capabilities in multiple images understanding (e.g., MM1 [2], mPLUG-Owl2 [3], MMICL [4]). These models should be included in the evaluation to provide a comprehensive understanding of their capabilities and how training strategy/training data/model designing affect the multimodal long-context understanding capabilities. 3. The experiment results indicate that some models struggle to follow instructions and perform poorly on several samples. It begs the question of whether these models inherently lack the multimodal long-context understanding or if their performance is hindered by unfamiliar patterns that might be improved with training on a few of data samples. In essence, the authors are encouraged to delve deeper into discussing which fundamental abilities, which preventing the models from effectively tackling multimodal long-context problems, are absent. [1] Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., & Gan, C. (2021, August). Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2). [2] McKinzie, B., Gan, Z., Fauconnier, J. P., Dodge, S., Zhang, B., Dufter, P., ... & Yang, Y. (2024). Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611. [3] Ye, Q., Xu, H., Ye, J., Yan, M., Liu, H., Qian, Q., ... & Zhou, J. (2023). mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257. [4] Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., ... & Chang, B. (2023). Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915.

Questions to authors

Refer to the Reasons To Reject.

Summary

The paper introduces MILEBENCH, a benchmark designed to assess the capabilities of Multimodal Large Language Models (MLLMs) in handling long-context scenarios involving multiple images and texts. This benchmark is motivated by the limitations of existing benchmarks that predominantly focus on short texts and single images, which do not adequately represent real-world multimodal tasks that require handling complex, lengthy contexts with multiple images. MILEBENCH includes two types of evaluations: diagnostic and realistic, aiming to thoroughly assess the performance of MLLMs in complex scenarios that mimic real-world applications. The benchmark tests 19 different models, highlighting that while the proprietary model GPT-4(Vision) excels, most open-source models struggle with these tasks.

Rating

6

Confidence

4

Ethics flag

2

Reasons to accept

1. The introduction of MILEBENCH addresses a significant gap in the evaluation of MLLMs by focusing on long-context multimodal scenarios, which are crucial for real-world applications but often overlooked in other benchmarks. 2. The paper does a good job of designing a variety of tasks that test different aspects of MLLMs, such as temporal and semantic understanding, across both realistic and diagnostic settings.

Reasons to reject

1. Majority of the curated benchmark comes from existing dataset, which may have data leakage and lead to unfairness and bias. Moreover, which part of the existing dataset did you extract the data from, training set, validation set, or test set? 2. While MILEBENCH incorporates a variety of tasks and sources, there is little discussion on the potential biases inherent in these sources or how they might affect the benchmark's generalizability.

Questions to authors

Please check the "Reasons To Reject"

Ethics concerns details

This paper release new datasets.

Summary

This paper presents a new benchmark, with a focus on multi-image scenarios. The benchmark mainly evaluates the long-context performance on the M-LLMs. The paper provides a comprehensive evaluation of several existing models, and shows that long-context capability is demanding for improvements.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

- The paper presents a comprehensive performance benchmarking. - Compared to prior studies, the proposed MileBench benchmark can be suitable for evaluating multimodal long-context abilities of MLLMs. - The evaluation taxonomy is clearly defined to evaluate two major categories: realistic and diagnostic evaluation. Each category includes a diverse set of evaluation tasks for evaluating the performance of multi-image scenarios. - The paper offers error analysis to help understand the challenging cases in the proposed dataset.

Reasons to reject

- The testing samples of the proposed dataset are mostly manually borrowed from existing datasets. There is no reliable pipeline to further collect and scale up. - It remains difficult to automatically evaluate these models which have free-form text outputs. The experiments only consider ROUGE-L and retrieval accuracy as the main metric. This may limit the long-context output scenarios.

Official Comment for Response

Thanks for the response, I would like to keep my score.

Response

Thanks for pointing out that the data sources are actually already listed in the appendix - it's my bad I didn't notice previously. Since I presented is originally as a reason to reject, I am raising the score to 7. I acknowledge the rest of the answers, however they don't particularly ease my concerns about insufficient data leakage discussion in the paper.

Thanks for the response

Thanks for your response. I would maintain the rating.

Thanks for the response

Thanks for your response. I would maintain the rating.