MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

Response to Reviewer g41s (Q1)

**In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with purple marks based on your comments.** Q1: some writing issues A1: Thank you for your correction! We have made the necessary changes and thoroughly reviewed the entire text again. Let us know if there is anything else we can refine or clarify further.

Response to Reviewer g41s (Q2)

Q2: Are there any duplicates between the samples in your dataset and the current multi graph dataset? A2: Compared to other multi-image benchmarks such as **MuirBench** [1] and **MilesBench** [2] , MMIU features test data that is highly distinct, establishing itself as an independently developed benchmark. To validate this, we calculate the visual similarity between the images in MMIU and those in MuirBench and MilesBench. Specifically, we use **LPIPS** [3] as the similarity metric. Many image editing works adopt LPIPS as a standard measure, and based on reported results, an LPIPS value below **0.1** is generally considered indicative of relatively similar images [4,5]. As shown in **Table 11 below**, when using a threshold of **0.1**, the image duplication rates between MMIU and MuirBench or MilesBench are only **0.1%** and **1.6%**, respectively. This demonstrates that MMIU is a highly independent and novel evaluation benchmark. Let me know if this aligns with your expectations! | **LPIPS ↓** | 0.1 | 0.2 | 0.3 | 0.4 | | -------------- | ---- | ---- | ---- | ----- | | **MuirBench** | 0.1% | 1.8% | 6.4% | 11.8% | | **MilesBench** | 1.6% | 3.3% | 7.8% | 12.9% | ​ Table11: LPIPS between MMIU and MuirBench, MilesBench [1] MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding [2] MileBench: Benchmarking MLLMs in Long Context [3] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric [4] Diffree: Text-guided shape free object inpainting with diffusion model [5] Inversion-Free Image Editing with Language-Guided Diffusion Models

Looking forward to your reply !

We would like to extend our heartfelt gratitude for the invaluable suggestions provided for this paper. In light of your feedback, we have diligently provided comprehensive and elaborate explanations. If you have any further inquiries, we are more than delighted to offer additional clarifications and conduct further experiments. We eagerly anticipate your response!

Response to Reviewer g41s (Q3)

Q3: Why do models without multi-imgae SFT... A3: Thanks to the reviewer for his suggestion. We miss this question before and now we give a detailed answer. For the original paper, InternVL2-Pro (only used video data for "multi-image SFT") and InternVL-Chat-1.5 surpassed some multi-image SFT models, but since the sizes of these two models (**78B and 26B**) are large, we add more models for comparison. Specifically, Below we comprehensively discuss the effect of multi-image data at different training stages. These contents are also involved in [Response to Reviewer bVMZ (Q5)](https://openreview.net/forum?id=WsgEWL8i0K&noteId=ZiG52vKojd) and we provide a more detailed analysis in **Appendix B.3: Discussion on the Effectiveness of Multi-Image Training**. We consider the two most common training stages: **pretraining** and **SFT (supervised fine-tuning)**. Based on the inclusion of multi-image data at these stages, we categorized models into four groups. As shown in **Table 12 below**, we find the following: 1. **Models exhibit some generalization to multi-image tasks without multi-image data**: Models trained without multi-image data, such as **Mini-InternVL1.5-chat**, still achieve reasonable performance on multi-image tasks, and even surpass the model trained with multi-image SFT: XGen-MM-Interleaved, which we attribute to their strong visual encoders leading to **excellent image understanding capabilities**. (e.g. Mini-InternVL1.5-4B even outperforms LLaVA-Next-34B on the multimodal comprehensive evaluation benchmark **MME**). However, even so, their performance is still lower than that of models trained with multi-image SFT such as Mantis, highlighting the importance of multi-image SFT 2. **Multi-image SFT consistently improves performance**: Regardless of whether multi-image pretraining is conducted, models like **LLaVA-Next-Interleave** and **XGen-MM-Phi3-Interleaved** show significant improvements when multi-image SFT is applied. 3. **Combining multi-image SFT with pretraining yields better results**: Models that incorporate multi-image data at both stages, such as **Mantis-idesfics2**, outperform those relying solely on SFT, such as **Mantis-SigCLIP**. 4. **Multi-image pretraining alone is insufficient**: Models that only perform multi-image pretraining, like **XGen-MM-Phi3-Single** and **idefics2**, underperform compared to those that only apply multi-image SFT (e.g. **Mantis-SigCLIP**), highlighting the importance of multi-image SFT. If you have further questions, please do not hesitate to let me know immediately. We sincerely hope that you will consider raising your rating and we will do our best to answer your questions. | **Model** | **Multi-image Pretrained** | **Multi-image SFT** | **Size** | **Semantic** | **Temporal** | **Spatial** | **Overall** | | ---------------------------- | -------------------------- | ------------------- | -------- | ------------ | ------------ | ----------- | ----------- | | Mini-InternVL1.5-chat | No | No | 2B | 30.4 | 36.2 | 26.5 | 30.5 | | Mini-InternVL1.5-chat | No | No | 4B | 32.2 | 36.8 | 28.5 | 32.1 | | InternVL1.5-chat | No | No | 26B | 33.4 | 45.2 | 35.5 | 37.4 | | LLaVA-Next | No | No | 7B | 23.0 | 24.1 | 20.3 | 22.2 | | LLaVA-Next-Interleave | No | Yes | 7B | 35.7 | 32.2 | 31.1 | 32.4 | | Mantis-SigCLIP | No | Yes | 8B | 53.6 | 38.4 | 35.7 | 42.6 | | xGen-MM-Phi3-Single [1] | Yes | No | 4B | 25.1 | 25.7 | 23.1 | 24.5 | | idefics2 | Yes | No | 8B | 31.6 | 20.9 | 29.2 | 27.8 | | InternVL2 | Yes | Only video | 8B | 35.2 | 39.5 | 31.2 | 34.8 | | Mantis-idsfics2 | Yes | Yes | 8B | 56.3 | 40.8 | 39.2 | 45.6 | | XGen-MM-Phi3-Interleaved [1] | Yes | Yes | 4B | 28.2 | 28.7 | 29.1 | 28.7 | Table12: Performance comparison of various models with multi-image training data in different stage on MMIU. [1] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Looking forward to your reply !

As the discussion period is nearing its end, we greatly value your feedback and are more than happy to provide further clarifications if needed. If you have any additional questions, please feel free to reach out immediately. We sincerely hope you will consider revising your rating, and we will do our utmost to address any concerns you may have.

Response to Reviewer JjzH (Q1)

**In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with orange marks based on your comments.** Q1: ... more specific insights into how different architectures impact ... A1: We sincerely thank the reviewer for the insightful comments. Indeed, architecture plays a significant role in influencing model performance. However, given the differences in training datasets for the tested models, it is challenging to conduct detailed controlled experiments. To address this, we select some of the best-performing models and analyze commonalities in their architectures that may contribute to better performance on multi-image tasks. **1) High resolution and dynamic resolution are crucial.** Research suggests that increasing resolution and incorporating dynamic resolution during training significantly improve image understanding performance (e.g., SPHINX [1]). This is because low resolutions tend to miss critical visual details, and training on a single resolution size can lead to image distortions or inefficiencies in training and inference. For multi-image tasks, we observe that top-performing models (e.g., **InternVL2-8B, Mantis-idefics2, LLaVA-Next-Interleave**) adopt high resolutions (448, 384, and 336, respectively) alongside dynamic resolution training strategies. This enables the models to better comprehend visual inputs. **2) A strong visual encoder makes a difference.** The **InternVL** series demonstrates outstanding performance on MMIU. For instance, both **Mini-InternVL-Chat-1.5-4B** and **XGen-MM-Interleave-4B** [2] use **Phi3-mini** [3] as their LLM, yet InternVL consistently performs better. Remarkably, **Mini-InternVL-Chat-1.5-4B** achieves performance on par with the larger **LLaVA-Next-Interleave-7B**. We attribute this partly to **InternViT**, the encoder used in InternVL models. InternViT is a powerful vision encoder that outperforms CLIP-based alternatives across multiple benchmarks. Thus, having a robust vision encoder is instrumental in enhancing performance on multi-image tasks. We provide a more detailed analysis in **Appendix B.3**. Thank you again for your constructive comments, which have helped us refine and deepen our analysis. [1] SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models [2] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models [3] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Response to Reviewer JjzH (Q2)

Q2: While MMIU successfully identifies gaps... A2: We sincerely thank the reviewer for the valuable suggestions. Indeed, MMIU does not involve training a specific model or proposing a new method to solve multi-image understanding tasks. However, training a model is not the only way to contribute. MMIU provides a significant contribution by testing a wide range of models on comprehensive multi-image tasks, conducting detailed analyses, and summarizing potential approaches to improve multi-image performance: 1. **Sufficient multi-image SFT greatly enhances performance:** For example, **Mantis-idesfics2**, after performing multi-image SFT on the base model **idefics2**, achieves a remarkable **17.8% accuracy improvement**. Even without multi-image pretraining, models like **Mantis-SigCLIP** still achieve a relatively high accuracy of **42.6%**, showing the effectiveness of multi-image SFT. 2. **Combining multi-image SFT with multi-image pretraining yields better results:** For instance, **Mantis-idesfics2** outperforms **Mantis-SigCLIP** because the base model of the former underwent interleaved image-text pretraining, resulting in higher accuracy. 3. **Multi-image pretraining alone may not be cost-effective:** Models like **idefics2** and **xGen-MM-Single**, despite undergoing multi-image pretraining, show suboptimal performance without sufficient multi-image SFT. This highlights the importance of coupling pretraining with fine-tuning for optimal results. 4. **High resolution and dynamic resolution matter:** Please refer to our response to **[Response to Reviewer JjzH (Q1)](https://openreview.net/forum?id=WsgEWL8i0K&noteId=GypD7Nx1i2)** for further details. 5. **A strong visual encoder is crucial:** Please refer to our response to **[Response to Reviewer JjzH (Q1)](https://openreview.net/forum?id=WsgEWL8i0K&noteId=GypD7Nx1i2)** for further discussion. We provide a more detailed explanation of these findings in **Appendix B.3**. Once again, we thank the reviewer for their insightful comments. We fully agree that proposing and validating solutions is also an important contribution, and we will prioritize this as part of our future work. Thank you for helping us improve the scope and depth of our analysis!

Response to Reviewer JjzH (Q3)

Q3: ... more granular examination of how and why certain models excel or struggle with specific relationship types A3: Thank you for your question. We conduct a statistical error case analysis on four representative models across different multi-image relationships, and we summarize the findings as follows: **1）Semantic tasks are more prone to perception errors** compared to temporal and spatial tasks. This is attributed to the greater diversity inherent in semantic tasks, such as image retrieval, which poses additional challenges for multi-image understanding. In contrast, temporal and spatial tasks typically involve images with inherent consistency, like changes in time or viewpoint, making them less susceptible to perception errors. **2）Reasoning errors are more common in temporal and spatial tasks.** Temporal tasks often require models to infer object dynamics over time (e.g., ordering tasks or next image prediction), while spatial tasks demand reasoning about spatial relationships and cognition (e.g., Image-Captioning-with-Spatial-Context). Current models consistently struggle with these complex reasoning requirements. **3）Spatial tasks are particularly affected by capacity-related errors.** Tasks requiring advanced visual understanding, such as 3D pose estimation, exhibit a significant number of errors caused by the model’s limited capacity. This is likely due to insufficient training for specialized visual challenges, resulting in irrelevant or incoherent responses. **4）Open-source models are more prone to perception errors than closed-source models.** This disparity is largely due to the capabilities of the visual encoder, which is critical for multi-image understanding. Enhancing the performance of visual encoders in open-source models could effectively reduce these perception errors. **5）Smaller open-source models often fail to follow instructions,** particularly in long-context multi-image tasks. Their limited language model size impairs their ability to process and understand complex multi-image inputs. This highlights the importance of improving LLM capacity, especially for tasks requiring long-context understanding, to achieve better performance in multi-image applications. We visualize the error types for different models under various image relationships in **Figure 9** and provide a detailed analysis in **Appendix B.4**.

Looking forward to your reply !

Thank you very much for your insightful suggestions on this paper. We have responded to each of these in great detail. And we kindly request that you consider updating the score accordingly. If you have any other questions, we are more than happy to provide additional clarification as well as experiments, and look forward to your reply !

Looking forward to your reply !

As the discussion period is nearing its end, we truly appreciate your feedback and are eager to provide any additional clarifications if necessary. Should you have any further questions, please don't hesitate to contact me right away. We sincerely hope you will reconsider your rating, and we will make every effort to address your concerns.

Response to Reviewer 4Ca4 (Q1)

**In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with green marks based on your comments.** Thank you for raising a series of insightful questions. Considering that GPT-4o was only used to generate part of the incorrect options in the dataset, we have summarized your concerns into the following three key questions。 - **Q1:** What measures were taken to ensure that the generated incorrect options are genuinely incorrect and sufficiently challenging. - **Q2:** Limited Discussion on Practical Applicability。 - **Q3:** How do you ensure the high quality of incorrect options generated by GPT-4o Q1: What measures were taken to ensure that the generated incorrect options are genuinely incorrect and sufficiently challenging. A1: In generating incorrect options, we employ a variety of task-specific methods, carefully designed to ensure the quality and challenge of the distractors: 1. **For tasks where the ground truth is a numerical value (e.g., object tracking):** We use Python scripts to introduce random perturbations to the ground truth values within a fixed standard deviation. This approach generates plausible but incorrect options that remain close to the original values, increasing the difficulty of the task. 2. **For tasks where the ground truth is a long text (e.g., multi-image captioning):** We use a **few-shot prompting strategy** with GPT-4o. The prompts are designed to help GPT-4o understand what constitutes a challenging incorrect option. For example, given a ground truth caption such as *"The woman and man were eating some street food,"* we provide examples of incorrect options with varying quality scores, as shown in **Table 9 [in the response to reviewer 4Ca4 (Q3)](https://openreview.net/forum?id=WsgEWL8i0K&noteId=a2Fc45rwoU)** . GPT-4o is then tasked with generating high-quality distractors based on these examples. Experiments (as discussed in our **A3 response**) and manual inspections confirm that this method produces high-quality incorrect options. Throughout this process, we tailor the generation of incorrect options to the unique characteristics of each task type. This ensures that the multiple-choice questions in MMIU are not **naive or trivial**. Our approach aligns with methodologies used in SEED-Bench [1] and MMT-Bench [2], which demonstrate that combining human design with LLM assistance reliably produces high-quality incorrect options. [1] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension [2] MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

Response to Reviewer 4Ca4 (Q2)

Q2: Limited Discussion on Practical Applicability... A2: Thank you for your valuable suggestion. We greatly appreciate your insights, which help us emphasize the practical relevance of MMIU. As shown in **Table 7 below**, we discuss this from four perspectives: 1. **Embodied Systems and Robotics** Unlike other benchmarks that focus primarily on semantic-related tasks, MMIU includes a rich set of spatial tasks that enable extensive exploration of the visual capabilities required for high-level planning in **embodied systems** and their applications in **robotics**. For example, in the **Egocentric Video Question Answering (EVQA)** task, **InternVL2** outperforms other LVLMs, demonstrating its potential for physical interactions with the environment. 2. **Autonomous Driving** MMIU also includes spatial tasks that are critical for **autonomous driving**, such as **3D depth estimation**, which provides vehicles with accurate, real-time spatial awareness and situational understanding in dynamic environments. Our evaluations show that most LVLMs face challenges in this task, highlighting their current limitations in spatial reasoning capabilities. 3. **Surveillance Systems** MMIU incorporates semantic tasks such as **Vehicle Re-Identification** and **Person Re-Identification**, which are essential for improving the effectiveness and capabilities of **surveillance systems**. In these tasks, **InternVL2** outperforms other LVLMs, including the closed-source model GPT-4o and the multi-image SFT model Mantis, showcasing its strong potential in the surveillance domain. 4. **Autonomous Control Applications** MMIU includes temporal tasks like **GUI app recognition** and **GUI next action prediction**, which have significant practical value for autonomous control systems relying on visual inputs. Our results show that **Claude3.5** demonstrates relatively better performance in these tasks compared to other LVLMs, highlighting its growth potential for future development. **Conclusion** MMIU serves as a comprehensive benchmark encompassing diverse tasks, providing critical insights into the strengths and limitations of current LVLMs across spatial, semantic, and temporal dimensions. By doing so, it contributes to driving advancements in real-world applications. We provide further details in **Appendix B.3: Practical Applicability Discussion**. Once again, thank you for your insightful suggestions, which have been instrumental in refining our analysis and highlighting the broader significance of our work. | Model | Egocentric Video Question Answering | 3D Depth Estimation | Vehicle Retrieval | Person Re-ID | GUI App Recognition | GUI Next Action Prediction | | ------------------ | ----------------------------------- | ------------------- | ----------------- | ------------ | ------------------- | -------------------------- | | GPT-4o | 57.5 | 24.0 | 68.0 | 61.5 | 73.5 | 46.5 | | Claude3.5-Sonnect | 53.5 | 23.5 | 79.0 | 77.5 | 88.5 | 55.0 | | Mantis-idefics2-8B | 54.5 | 23.5 | 70.5 | 57.5 | 78.0 | 34.0 | | InternVL2-Pro | 63.0 | 25.5 | 83.5 | 82.0 | 91.5 | 40.5 | ​ Table7: The performance of some models on MMIU's practical tasks

Response to Reviewer 4Ca4 (Q3)

Q3: How do you ensure the high quality of incorrect options generated by GPT-4o A3: On the one hand, we explain that although GPT-4o has only moderate multi-image reasoning capabilities, it can still generate competitive incorrect options using the approach described in **A1**: 1. **The task is to generate incorrect options, not solve the problem:** Instead of requiring GPT-4o to solve the question, we provide it with the question and the correct answer to generate incorrect options. By presenting the correct answer, the difficulty of understanding the question is effectively reduced. This is because GPT-4o can identify which image or region the question focuses on and then perform targeted reasoning based on the correct answer. Once the question is understood, GPT-4o can generate incorrect options by reasoning about semantic differences between the options and the images. 2. **Task-specific few-shot prompts for incorrect option generation:** We design specific few-shot prompts tailored to different tasks to enhance the quality of incorrect options. For example, given a ground truth caption such as *"The woman and man were eating some street food,"* we provide examples of incorrect options with varying quality scores, as shown in **Table 9 below**. GPT-4o is then tasked with generating high-quality distractors based on these examples. This targeted approach allows GPT-4o to produce challenging and semantically relevant incorrect options, even with its moderate multi-image reasoning abilities. On the other hand, We demonstrate that the incorrect options in MMIU are of high quality through two experiments and manual evaluation: 1. **Performance without Visual Input:** As shown in **Table 3** of the main text, when images are removed and LLMs are tasked with answering questions using only the questions and options, the overall accuracy of multiple LLMs approaches random guessing. This indicates that it is difficult to distinguish between correct and incorrect options solely based on the options, demonstrating that the incorrect options are sufficiently challenging. 2. **Semantic Similarity Analysis:** We randomly select **500 samples** whose incorrect options are generated by GPT-4o. Using OpenAI’s **text-embedding-3-small** model, we extract embeddings for both the correct and incorrect options and compute their cosine similarity. As a baseline, we randomly select incorrect options from other samples within the same task, ensuring that these incorrect options differ from the correct option of the current sample. As shown in **Table 8 below**, the results show that the semantic similarity between GPT-generated incorrect options and the correct options is significantly higher than the baseline. This indicates that GPT-generated incorrect options are more semantically similar to the correct options, making them more challenging. | Similarity | Semantic | Temporal | Spatial | | ---------- | -------- | -------- | ------- | | GPT | 0.64 | 0.56 | 0.54 | | Random | 0.21 | 0.18 | 0.15 | ​ Table8: Comparison of semantic similarity between wrong options and correct options

Response to Reviewer 4Ca4 (Q3)

3. **Human Evaluation of Incorrect Options:** We use the **500 samples** selected in experiment 2 and recruit **5 senior undergraduate students** to evaluate the quality of the incorrect options. Each sample is assigned to all 5 annotators, who score the quality of the incorrect options based on the provided guidelines (examples shown in **Table 9 below**). Annotators evaluate the options on a standardized scale. The cost of this manual evaluation is detailed in **Table 10 below**. The results show an average score of **0.93 (after normalization)** across all annotators, with no samples receiving a score of **0**. Additionally, the average variance among annotators’ scores is **0.06**, indicating strong agreement among annotators. This consistency demonstrates that GPT-4o-generated incorrect options in MMIU are widely considered to be challenging and effective distractors. | Images | Question Description | Correct Answer | Incorrect Option | Quality of Incorrect Options | Reason | | ----------- | ------------------------------------------------------ | ---------------------------------------------- | ---------------------------------------------- | ---------------------------- | ------------------------------------------------------------ | | Images list | Select the option with describe these images correctly | The woman and man were eating some street food | The man and woman were cooking dinner at home | Moderate (1) | The option is indeed wrong, but the behavior and scenario have changed | | | | | The couple were eating some food in the street | Poor (0) | The option is same with the correct option | | | | | The couple were eating some food in the park | High (2) | The option is indeed wrong, and only scenario has changed | ​ Table9: Example of incorrect option quality annotation | Samples | Avg. time (per sample) | Num. People | Background | Cost | Total Cost | | ------- | ---------------------- | ----------- | --------------- | -------- | ---------- | | 500 | 30s-1min | 5 | College Student | 8USD / h | 200USD | ​ Table10: Cost of manual inspection

Looking forward to your reply !

We express our sincere appreciation for the valuable suggestions regarding this paper. In response, we have provided thorough and detailed explanations. And we kindly request that you consider updating the score accordingly. If there are any further questions, we are delighted to offer additional clarifications and conduct further experiments!

Looking forward to your reply !

As the discussion period is nearing its end, we sincerely thank you for your valuable feedback and are happy to offer any further explanations if needed. If you have additional questions, please feel free to reach out at any time. We genuinely hope you will reconsider your rating, and we are committed to addressing any issues you may have.

Response to Reviewer avPS (Q1)

**In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with red marks based on your comments.** Q1: My bigger issue about this paper is regarding the presentation of experimental results ... A2: Thank you for your valuable suggestions regarding our writing. We implement numerous revisions in the resubmitted paper to improve its readability and clarity: 1. **Revised Main Results Table** We replace the main results table in the main text with the average accuracy of LVLMs across seven types of image relationships. In **Section 4.2**, we analyze the overall trends across different models, while in **Section 4.3.1**, we provide a detailed performance analysis of different models under varying image relationships. 2. **Enhanced Focus in Figure 4** - We highlight the clustering results of the **task map** from the original **Figure 4(a)** by emphasizing the shared characteristics of each cluster. These details, along with annotations, are now presented in **Appendix Figure 11**, as our primary focus is the analysis of model performance based on clustering. - For the original **Figure 4(b)**, which presents model performance, we simplify the heatmap to create the current **Figure 4** by categorizing tasks into clusters and grouping models. This improves readability while maintaining essential insights. Additional analyses are provided in **Appendix C.2**. 3. **Separated Small Tables for Independent Analysis** For smaller tables, such as **Appendix Table 16** and **Table 17**, we extract key results (e.g., representative models' performance on different image relationships or tasks) and conduct more detailed discussions in **Appendix B.3**. 4. **Expanded Visualizations for Important Content** For example, we revise **Appendix Figure 9** to include more detailed visualizations categorized by image relationships. This refinement addresses the broad scope of **Section 4.4 (Error Analysis)** by providing a more targeted and visually interpretable breakdown, thereby improving the presentation of key findings. We appreciate your constructive feedback, which significantly contributes to enhancing the clarity and presentation of our work.

Response to Reviewer avPS (Q2)

Q2: Potential position bias in the unanswerable set is unaddressed... A2: Thank you for your question, and we apologize for any confusion caused. In fact, we consider this aspect not only when designing the unanswerable set but also throughout the overall benchmark. The correct options in all tasks are always randomized in order. As shown in **Table 6 below**, for the unanswerable set, the placement of the unanswerable option is also consistently randomized. This approach minimizes the potential **position bias** issue you mention to the greatest extent possible. Additionally, we provide further clarification in **Section 4.1 (Evaluation Method)** to ensure this point is clearly addressed. Thank you again for raising this important consideration. | Position | A | B | C | D | | -------- | ----- | ----- | ----- | ----- | | Portion | 24.6% | 24.7% | 25.6% | 25.1% | ​ Table 6: The probability of unanswerable options appearing in different positions is almost equal

Response to Reviewer avPS (Q2)

Q2: Is there any baseline performance by human A2: We would like to thank the reviewer for their insightful comments. These suggestions have been very helpful, and we have given them detailed consideration. Below, we address each point in turn. 1. **Recruitment of Human Evaluators**: We continue to invite the five students employed at **[Response to Reviewer 4Ca4 (Q3)](https://openreview.net/forum?id=WsgEWL8i0K&noteId=a2Fc45rwoU)** to continue the task of answering questions. Given time constraints, we select 41 tasks out of the original 52 which are suitable for human evaluation. Because some tasks in MMIU, such as tracking-related tasks, are more suited for machines to answer, while human evaluation is overly labor-intensive. From each task, we sample 10 instances, resulting in a total of 410 samples for human evaluation. For each sample, we assign 5 students to answer. The average accuracy across 7 relationships is then calculated, as shown in **Table 2 below**. The manual effort involved in this process is recorded in **Table 3 below**. 2. **Evaluation Results**: The results show that the current model performs relatively well for Low-Level image relationships and High-Level (subjective) tasks, approaching human scores. However, the overall performance still lags behind human performance, particularly in spatial relationships, where the gap between model and human ratings is most pronounced. This highlights the current limitations of the model in handling multi-image tasks. 3. **Future Work**: We will expand the sample size for further human evaluation. For some visual tasks, we plan to invite more senior annotators and use **specialized visual models** (e.g., Depth Anything) to test the accuracy of these tasks. This will provide a more comprehensive evaluation of MMIU's **theoretical upper bounds**. Once again, thank you for your valuable feedback. If the reviewer has any further questions or concerns, please do not hesitate to ask, and we will do our best to address them. | Samples | Avg. time (per sample) | Num. People | Background | Cost | Total Cost | | ------- | ---------------------- | ----------- | --------------- | -------- | ---------- | | 420 | 1min-2min | 5 | College Student | 8USD / h | 384 USD | ​ Table2: The cost of human annotators. | Overall | Overall | Discrete | Continuous | Low-level | High-level(sub) | High-level(obj) | Two-D | Three-D | | ------- | ------- | -------- | ---------- | --------- | --------------- | --------------- | ----- | ------- | | Human | 92.8 | 89.5 | 92.8 | 98.5 | 96.6 | 94.6 | 88.0 | 92.0 | | GPT-4o | 55.5 | 58.2 | 53.7 | 84.0 | 69.2 | 57.5 | 41.7 | 55.4 | Table 3: Comparison of human results and GPT-4 results across 7 image relationships.

Response to Reviewer bVMZ (Q1)

**In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with blue marks based on your comments.** Q1: MMIU ... This may require a lot of computing resources A1: To mitigate the computational cost caused by the large number of tasks, similar to MathVista [1], we construct a smaller test set, **testmini**, by randomly sampling 20 examples from each of the 52 tasks, resulting in a total of 1,040 examples. While preserving the diversity of MMIU, our experiments show that the results on **testmini** effectively reflect those on the full test set. These findings appear in **Table 3** in the main text and **Table 15** in the appendix, where we report results on both the full test set and **testmini**. Additionally, we perform a quantitative analysis of the testing cost for **testmini**. As shown in **Table 1 below**, most models complete the evaluation on a single A100 GPU within one hour. Even for the closed-source model, GPT-4o, the testing cost remains manageable, requiring approximately $10 and one hour to complete. Moving forward, we plan to evaluate more models on **testmini** and provide a **convenient interface** to enable the community to test and compare models on MMIU seamlessly. Once again, thank you for your thoughtful suggestions, which are instrumental in improving our work. | Model | Size | BS（per device） | Resources | Times | Memory | | --------------------- | ---- | ---------------- | ------------- | ----- | -------- | | GPT-4o | - | 4 | USD 8.57 | 0.51h | - | | LLaVA-Next-Interleave | 7B | 3 | 1 x A100-80GB | 0.04h | 60381MiB | | Mantis-idefics2 | 8B | 6 | 1 x A100-80GB | 0.15h | 73896MiB | | XGen-MM | 4B | 7 | 1 x A100-80GB | 0.14h | 75082MiB | | InternVL-Chat-V1.5 | 26B | 1 | 1 x A100-80GB | 0.89h | 71619MiB | | InternVL2 | 8B | 1 | 1 x A100-80GB | 0.97h | 79292MiB | | LLaVA-1.5 | 7B | 4 | 1 x A100-80GB | 0.02h | 64462MiB | ​ Table1: Resource consumption of some models in MMIU's testmini set [1] MathVista: Evaluating Math Reasoning in Visual Contexts

Response to Reviewer bVMZ (Q2)

Q2: The paper shows that ... lack of practical significance A2: Thank you for your constructive feedback. We have carefully considered your comments and would like to address them in detail below: **1）Multi-Image Training for Temporal and Spatial Relationships in LVLMs** We acknowledge that earlier single-image models like LLaVA and MiniGPT-4 primarily focused on modeling semantic relationships. However, with the advancement of large vision-language models (LVLMs), many now incorporate multi-image data during training to capture temporal and spatial relationships. For instance: - **Mantis** utilizes **Mantis-Instruct** [1] for multi-image supervised fine-tuning (SFT), which includes datasets like VIST [2] and NExT-QA [3], emphasizing temporal relationships. - **LLaVA-Next-Interleave** employs **M4-Instruct** [4] for multi-image SFT, incorporating embodied tasks such as 3D grounding that highlight spatial reasoning. - **InternVL2** utilizes **OmniCorpus** [5] during pretraining, a dataset with 10 billion interleaved image-text pairs, encompassing diverse semantic, temporal, and spatial relationships. Additionally, in its Stage 2 training, it incorporates video data (e.g., **EgoTaskQA** [6]), enabling it to effectively handle tasks involving both spatial and temporal relationships. These advancements demonstrate the growing focus on multi-image capabilities, aligning with the evolving needs of LVLMs. **2) Practical Significance of MMIU Task Design** We also emphasize the practical value of MMIU’s task design. Beyond general high-level semantic tasks, MMIU incorporates tasks with real-world applications, relevant to fields like robotics and autonomous driving. As shown in **Table 2 below**, we analyzed selected tasks and model performances, highlighting the following: - MMIU evaluates high-level visual capabilities for **embodied systems**, such as robotics. For example, **InternVL2-Pro** demonstrates strong potential in egocentric tasks like EVQA. - MMIU includes spatial reasoning tasks critical for **autonomous driving**. Results reveal that current LVLMs face challenges in tasks like 3D depth estimation. - MMIU addresses semantic tasks relevant to **surveillance applications**. **InternVL2-Pro** excels in Vehicle and Person Re-Identification, outperforming other LVLMs in these scenarios. - MMIU incorporates temporal tasks for **autonomous control**, where **Claude 3.5** shows potential in GUI-related applications. We provide a more detailed discussion of these findings in **Appendix B.3: Practical Applicability Discussion**. Once again, thank you for your insightful suggestions, which have been instrumental in refining our analysis and emphasizing the broader significance of our work. | Model | Egocentric Video Question Answering | 3D Depth Estimation | Vehicle Retrieval | Person Re-ID | GUI App Recognition | GUI Next Action Prediction | | ------------------ | ----------------------------------- | ------------------- | ----------------- | ------------ | ------------------- | -------------------------- | | GPT-4o | 57.5 | 24.0 | 68.0 | 61.5 | 73.5 | 46.5 | | Claude3.5-Sonnect | 53.5 | 23.5 | 79.0 | 77.5 | 88.5 | 55.0 | | Mantis-idefics2-8B | 54.5 | 23.5 | 70.5 | 57.5 | 78.0 | 34.0 | | InternVL2-Pro | 63.0 | 25.5 | 83.5 | 82.0 | 91.5 | 40.5 | ​ Table2: The performance of some models on MMIU's practical tasks [1] Mantis: Interleaved Multi-Image Instruction Tuning [2] Visual Storytelling [3] NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions [4] LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models [5] OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text [6] EgoTaskQA: Understanding Human Tasks in Egocentric Videos

Response to Reviewer bVMZ (Q3)

Q3: The authors ... data processing techniques used A3: Thank you for pointing out this issue. We have supplemented our explanation with more detailed descriptions as follows: In **Section 3.2**, we outlined the general process of data collection, which involves the following steps: 1. **Defining tasks based on image relationships.** We first define specific tasks corresponding to the relationships among images. Detailed explanations of these tasks and their descriptions are provided in **Table 7** in the Appendix. 2. **Identifying datasets that reflect the tasks.** We then identify datasets suitable for each defined task, primarily through manual collection from sources such as Kaggle, GitHub, and Papers with Code. The specific **data sources** for each task are listed in **Table 7** of the Appendix. 3. **Transforming datasets into multiple-choice questions.** Next, we convert the collected datasets into a unified multiple-choice format: - First, the datasets are standardized into a unified metadata format, with an example provided in **Table 9** of the Appendix. - Then, we generate incorrect options based on the ground truth. For tasks where the ground truth is numerical (e.g., object detection), we use algorithmic perturbations to create distractors. For tasks involving text generation (e.g., captions), we employ a few-shot prompting approach with GPT-4o to generate high-quality distractors. For instance, given the ground truth "The woman and man were eating some street food," we provide three examples of distractors with different quality scores in **Table 9 in the [Response to Reviewer 4Ca4 (Q3)](https://openreview.net/forum?id=WsgEWL8i0K&noteId=a2Fc45rwoU)**, guiding GPT-4o to generate high-quality incorrect options. Detailed descriptions of the key steps in data generation, such as metadata standardization and distractor generation, are provided in **Appendix A.4**. We hope this additional information clarifies our data collection and processing methodology. Thank you again for your valuable feedback, which has helped us improve the clarity of our work.

Response to Reviewer bVMZ (Q4)

Q4: ...can you give some performance of pure language models. A4: Thank you for your suggestions. We have supplemented this part with additional experiments, as detailed below. 1. **Performance without Visual Input** As shown in **Table 3 below**, when images are excluded and only questions and options are provided, the scores of all LLMs are nearly equivalent to random guessing. This suggests that the distractors generated by GPT-4o are sufficiently challenging and meaningful. Furthermore, it highlights that MMIU tasks require visual information, demonstrating their distinct multimodal characteristics. 2. **Caption-Based Reasoning** We also explore an approach where GPT-4o extracted captions for each input image, and then LLMs were tasked with answering the questions based on the captions and options. As shown in **Table 4 below**, this approach led to significant improvement compared to random guessing. However, it still falls short compared to direct inference using LVLMs. For instance, GPT-4o’s direct inference achieves an overall accuracy of **55.5%**, outperforming the caption-based reasoning accuracy of **48.7%**. This finding underscores that accurately understanding the semantic information in images is essential for multi-image understanding tasks, while some tasks in MMIU require more **low-level visual information** to be effectively solved. We provide a more detailed analysis of these findings in **Appendix B.3**. Thank you again for your valuable feedback, which has helped enhance the depth and clarity of our evaluation. | Model | Temporal | Semantic | Spatial | Overall | | ----------- | -------- | -------- | ------- | ------- | | Random | 24.0 | 30.9 | 26.6 | 27.1 | | GPT-4o | 28.2 | 16.1 | 25.7 | 23.1 | | Llama3.1 | 29.3 | 31.1 | 28.3 | 29.5 | | Qwen2.5 | 28.6 | 35.3 | 28.5 | 30.9 | | InternLM2.5 | 32.2 | 31.4 | 31.2 | 31.5 | ​ Table3: Some LLMs performance on MMIU | Model | Temporal | Semantic | Spatial | Overall | | ----------- | -------- | -------- | ------- | ------- | | GPT-4o | 53.9 | 53.8 | 42.5 | 48.7 | | Llama3.1 | 40.0 | 43.1 | 35.2 | 39.2 | | Qwen2.5 | 44.4 | 45.2 | 34.2 | 40.3 | | InternLM2.5 | 42.3 | 42.1 | 33.2 | 38.2 | ​ Table4: Some LLMs with caption generated by GPT-4o performance on MMIU

Response to Reviewer bVMZ (Q5)

Q5: ... add interleave data or video data in the pre-training or post-training stage. A5: Your question is highly insightful, and we have conducted additional tests with a broader range of models to analyze the role of multi-image data at different training stages. We consider the two most common training stages: **pretraining** and **SFT (supervised fine-tuning)**. Based on the inclusion of multi-image data at these stages, we categorized models into four groups. As shown in **Table 5 below**, we find the following: 1. **Multi-image SFT consistently improves performance**: Regardless of whether multi-image pretraining is conducted, models like **LLaVA-Next-Interleave** and **XGen-MM-Phi3-Interleaved** show significant improvements when multi-image SFT is applied. 2. **Combining multi-image SFT with pretraining yields better results**: Models that incorporate multi-image data at both stages, such as **Mantis-idesfics2**, outperform those relying solely on SFT, such as **Mantis-SigCLIP**. 3. **Multi-image pretraining alone is insufficient**: Models that only perform multi-image pretraining, like **XGen-MM-Phi3-Single** and **idefics2**, underperform compared to those that only apply multi-image SFT (e.g. **Mantis-SigCLIP**), highlighting the importance of multi-image SFT. 4. **Models exhibit some generalization to multi-image tasks without multi-image data**: Models trained without multi-image data, such as **Mini-InternVL1.5**, still achieve reasonable performance on multi-image tasks, attributed to their strong visual encoders. However, their performance lags behind models with multi-image SFT. We provide a more detailed analysis in **Appendix B.3: Discussion on the Effectiveness of Multi-Image Training**. Thank you for your valuable feedback, which helped us deepen our analysis of multi-image training effectiveness. | **Model** | **Multi-image Pretrained** | **Multi-image SFT** | **Size** | **Semantic** | **Temporal** | **Spatial** | **Overall** | | ---------------------------- | -------------------------- | ------------------- | -------- | ------------ | ------------ | ----------- | ----------- | | Mini-InternVL1.5-chat | No | No | 2B | 30.4 | 36.2 | 26.5 | 30.5 | | Mini-InternVL1.5-chat | No | No | 4B | 32.2 | 36.8 | 28.5 | 32.1 | | InternVL1.5-chat | No | No | 26B | 33.4 | 45.2 | 35.5 | 37.4 | | LLaVA-Next | No | No | 7B | 23.0 | 24.1 | 20.3 | 22.2 | | LLaVA-Next-Interleave | No | Yes | 7B | 35.7 | 32.2 | 31.1 | 32.4 | | Mantis-SigCLIP | No | Yes | 8B | 53.6 | 38.4 | 35.7 | 42.6 | | xGen-MM-Phi3-Single [1] | Yes | No | 4B | 25.1 | 25.7 | 23.1 | 24.5 | | idefics2 | Yes | No | 8B | 31.6 | 20.9 | 29.2 | 27.8 | | InternVL2 | Yes | Only video | 8B | 35.2 | 39.5 | 31.2 | 34.8 | | Mantis-idsfics2 | Yes | Yes | 8B | 56.3 | 40.8 | 39.2 | 45.6 | | XGen-MM-Phi3-Interleaved [1] | Yes | Yes | 4B | 28.2 | 28.7 | 29.1 | 28.7 | ​ Table5: Performance comparison of various models with multi-image training data in different stage on MMIU. [1] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Response to Reviewer bVMZ

Thank you for your feedback. We have updated the latest version of the paper with additional content based on your suggestions, including the **Testmini settings** and **an analysis of the role of multi-image data at different training stages** (please refer to the **[Paper Update](https://openreview.net/forum?id=WsgEWL8i0K&noteId=XgIP7czmVN)** for specifics). For models that have undergone multi-image training, such as Gemini-Flash and Qwen2-VL, we have now tested over **30 LVLMs**, which provides a more comprehensive evaluation compared to other benchmarks. This includes models like **LLaVA-Interleave** and **InternVL2**, both of which have been trained with multi-image and video data. Additionally, we have included more recent models such as **XGen-MM**, **LLaVA-Interleave**, and **Mini-InternVL**. Absolutely, We plan to test even more models and provide convenient testing interfaces for the community to test on their own in the future. If you have any further questions, please don’t hesitate to let us know. We will make every effort to address them. And we kindly request that you consider updating the score accordingly. Thank you again for your valuable suggestions, which we believe have significantly enhanced the quality of the paper.

Response to Reviewer bVMZ

We would like to express our sincere gratitude for your insightful feedback. In fact, we have already conducted tests on **Gemini1.5Flash**, as well as on **InternVL2**, which is similar to **Qwen-VL2**. Both models have been extensively trained on **video data** and **interleaved image-and-text datasets**. The detailed results are summarized in **Table 1** below. Notably, We have evaluated **over 30 LVLMs and LLMs** on MMIU, compared to other benchmark papers, we think this is a fairly comprehensive evaluation. As for Gemini and InternVL: **Gemini1.5Flash**, **InternVL2-Pro**, and **InternVL2-8B** have achieved superior performance compared to other models, highlighting the importance of **multi-image training**. However, it is also important to note that these models still fall short when compared to **human scores** (referencing **[Response to Reviewer avPS (Q2)](https://openreview.net/forum?id=WsgEWL8i0K&noteId=mcGk8PFa6Z))**. This suggests that there is still significant room for improvement in the models' **multi-image capabilities**. In **Appendix B.3**, we provide a detailed analysis of both **architectural** and **data-related approaches** to enhance multi-image performance. Additionally, in **Appendix B.4**, we include a thorough **error case analysis**. We hope these findings can contribute to the development of more advanced multi-image models. Given the contributions of our work and the insights shared, we kindly ask if you might consider raising your score accordingly. If you have any further questions, we would be more than happy to provide additional clarifications. | Model | Overall | Discrete | Continuous | Low-level | High-level-sub | High-level-obj | Two-D | Three-D | |----------------|---------|----------|------------|-----------|----------------|----------------|-------|---------| | Gemini1.5-Flash | 54.5 | 54.2 | 50.1 | 76.1 | 63.9 | 64.9 | 43.3 | 43.0 | | InternVL2-Pro | 49.8 | 53.8 | 46.3 | 72.7 | 70.6 | 58.5 | 38.1 | 42.1 | | InternVL2-8B | 34.0 | 34.2 | 43.4 | 36.7 | 47.3 | 32.1 | 30.0 | 32.2 | | Human | **92.8** | **89.5** | **92.8** | **98.5** | **96.6** | **94.6** | **88.0** | **92.0** | Table 1: Some model performance and human performance on MMIU

Looking forward to your reply !

As the discussion period is nearing its end, we would like to sincerely thank you for your valuable feedback. If you have any further questions, please do not hesitate to contact me immediately. We are fully committed to addressing any concerns you may have.