Summary
The paper presents a novel approach to reducing redundancy in visual tokens within MLLMs by introducing the Visual Context Compressor (VCC) and Stage-wise MLLM Training. The VCC uses simple average pooling to compress visual tokens during training, enhancing efficiency without sacrificing performance. The Stage-wise MLLM Training progressively reduces compression as training proceeds, ensuring no loss of information during testing. The proposed methods demonstrate significant improvements in training efficiency and performance across multiple benchmarks.
Strengths
1. The paper addresses an underexplored area in MLLMs by focusing on the redundancy of visual tokens and introducing effective compression techniques.
2. The proposed methods achieve significant reductions in training costs while maintaining or improving performance on various benchmarks.
3. The paper provides thorough experimental validation across multiple benchmarks, demonstrating the effectiveness of the proposed methods.
Weaknesses
1. While the paper shows improvements across several benchmarks, the evaluation is primarily focused on visual question answering tasks. Additional evaluation on other multi-modal tasks could strengthen the claims.
2. The paper mentions potential information loss due to compression but does not provide a detailed analysis/visualization of how this might affect tasks requiring dense visual information.
3. This paper lacks some essential experiments. (1) The paper does not explore the effect of using larger input resolutions. For example, evaluating the method with 448×448 input images, which contain 2.25 times more visual tokens than the 336×336 input, could provide insights into the method's scalability. Designing a compression setting with a ratio of 225% for this input size and comparing it with the original LLaVA setting (compression ratio of 100%) would be valuable. (2) Testing the method on larger models such as LLaVA-13B or models with more input visual tokens (e.g., LLaVA-Next) could solidify the experimental section and demonstrate the robustness of the proposed approach.
4. The best stage-wise MLLM training scheme looks difficult to transfer to other model & data settings. The training scheme has too many options and variables. If the model size and dataset size increase significantly, the time cost and computation cost for finding the best scheme can become prohibitively large.
Questions
Please see the weakness.
Minor issues:
1. What does the #Tokens mean in the tables? Is it inversely proportional to CR?
2. How do you compute the average performance in Table 5?
3. Some suggestions for future improvements: (1) multi-stage compression: use different compressors’ strides in the different positions of the LLM. For example, layer 1-3: stride=1; layer 4-12: stride=2; layer 13-24: stride: 4. (2) extend this technique to other modalities, such as video understanding.
Limitations
The authors have adequately discussed the limitations.