InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

Summary

This paper introduces a large vision-language model named InternLM-XComposer2-4KHD, which is designed to process images up to 4K HD resolution. The model incorporates a dynamic resolution adaptation mechanism that maintains the aspect ratios of input images, thus enhancing the model's ability to understand complex image-text interactions. This approach allows for scalable resolution adjustments during training, supporting a wide range of resolutions. The model even outperforms well-known GPT-4V and Gemini Pro in 10 out of 16 benchmarks.

Strengths

1. This paper introduces a substantial enhancement in the field of large vision-language models. It demonstrates significant performance improvements across multiple benchmarks, outperforming well-known models such as GPT-4V and Gemini Pro, thereby potentially exerting a broad impact on the field. 2. The dynamic resolution design adopted in this study is particularly noteworthy. It enables the model to adapt to a variety of image resolutions while preserving critical detail. 3. The writing in the paper is commendable, with informative charts and examples that effectively showcase the model's capabilities.

Weaknesses

1. The paper frequently mentions training with 4K resolution, yet it lacks optimizations for accelerating high-resolution training. The heavy reliance on substantial computational resources limits the broader applicability of the proposed methods. 2. There is a lack of analysis regarding the inference efficiency of the model. While the paper validates that high-resolution training significantly enhances performance for text-based vision-language tasks, it does not adequately address the computational costs incurred by high-resolution inputs. This omission makes it challenging to assess whether the additional computational expenses are justified by the performance gains. 3. The introduction repetitively emphasizes adaptability to 4K resolution without providing clear problem statements or motivational examples. For instance, the discussion on "variability in patch layouts" (Line 52) lacks an intuitive explanation, and it would benefit from specific examples illustrating this variability. Besides, the paper highlights at Line 56 a critical aspect concerning the model's performance at resolutions beyond 4K with 'even when...'. It remains unclear whether this suggests that traditional methods trained at 4K resolution might fail to gain additional performance at lower resolutions. Clarifying this ambiguity is crucial for understanding the challenges the study addresses, reducing the need for readers to infer details solely from experimental results. 4. This work notably advances vision-language understanding by enhancing traditional OCR capabilities. However, it prompts inquiries about the model's proficiency with complex visual content. The examples presented do not conclusively demonstrate the model's ability to accurately interpret intricate visuals such as hand-written mathematical formulas, circuit diagrams, or chemical structures. These types of content demand detailed parsing for complex problem-solving, an aspect not fully illustrated by the simpler cases in the appendices. While these considerations may exceed the current study's scope, they do not impact the overall evaluation for acceptance.

Questions

1. The substantial computational resources required for training at 4K resolutions and the lack of detailed analysis on inference efficiency are main concerns. 2. The paper needs clearer problem statements and motivational examples. 3. The paper demonstrates enhancements in traditional OCR capabilities but leaves some uncertainty regarding the model's ability to process more complex visual content. If the authors can offer further insights or evidence, it would significantly strengthen the paper’s impact and might merit a higher evaluation.

Rating

5

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

The paper addresses its limitations and potential negative societal impacts in the appendices. The explanation of limitations lacks depth. For more detailed concerns, please refer to the questions section.

Summary

The paper presents InternLM-XComposer2-4KHD, a high-resolution MLLM that performs better than existing methods on various benchmarks.

Strengths

1. The paper is well written and easy to understand. 2. The performance of InternLM-XComposer2-4KHD is good, which is valided on various benchmarks.

Weaknesses

1. The paper has limited novelty. The high-resolution solution is not new. Starting from Monkey, there are a lot of works that adopt the similar idea, e.g., LLaVA-NeXT and InternVL-1.5. Compared to these methods, the technical contribution of the paper is trivial. 2. From the experimental results, I cannot tell whether the proposed high-resolution solution is better than other solutions. Since the setting of the paper are not comparable to others, it seems that the data plays a more significant role. Maybe more comparisons or ablations should be provided. 3. There still be a lot of MLLMs that have similar settings to IXC2-4KHD, e.g., InternVL-1.5. Why not compare with these models in Tab 3-4. 4. The implementation details are not clear. It's still confusing for me that how much data is used for training. In Tab 3, the previsous SoTA LLaVA-NeXT uses less than 2M data. So It's not clear whether the comparison is fair.

Questions

My main concern is the novelty and the experimental setting. The main idea has been widely explored in previous works.

Rating

3

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

See weakness.

Summary

This paper aims to explore the high resolution scenes of multimodal large language models. The authors find that the performances are largely improved when the model is equipped with 4K resolution. This finding will greatly inspire subsequent research works, which is of great value to the research community. The experimental results in the popular multimodal datasets, such as MME and MathVista, are impressive. The proposed dynamic resolution and automatic patch configuration will become a common approach, due to its simplicity and effectiveness.

Strengths

1. The motivation sounds good. High resolution is the inevitable development direction of multimodal large language models, and we urgently need to know what effects high resolution will bring to model training and performance. This paper stands on a high ground. 2. The method looks simple yet effective. Dynamic resolution and automatic patch configuration could not only maintain the aspect ratio of the image, but also indirectly enlarge the resolution of the image. 3. The experimental results on many datasets are impressive. The proposed method achieves SOTA results in 6 of the 16 benchmarks with only 8B parameters. 4. The writing of this paper is good, and the structure is easy to follow.

Weaknesses

1. Considering that high resolution is a very meaningful direction, it is suggested that the authors provide more details of the training to help other researchers, such as the loss figure of the training. Because training at high resolution is expensive, releasing more intermediate results from training will help researchers conduct further studies. 2. The current method seems to largely increase the number of visual tokens while increasing the resolution. One question is whether it is reasonable to use so such many visual tokens for ONE image. I would love to hear the authors' thoughts on this point. This is an open question. 3. It is suggested that the authors could enlarge the LLM parameters synchronously in subsequent studies if conditions permit, which will make the study more complete. It seems that image resolution and LLM parameters are the two most important aspects affecting model performance, in addition to the training data of course. 4. I am curious about the generalization performance of the proposed model on multimodal video evaluation dataset, such as Video-MME [1], which applies to both image MLLMs, i.e., generalizing to multiple images, and video MLLMs. The sub-figures in the split-patch strategy has some similarities with video frames. It would be great if the authors could update the results of the Video-MME into the paper. [1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. Arxiv, 2024.

Questions

None

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

None