DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs

Summary

The authors present a new architecture StackFormer - an effective and simple way to infuse fine-grained visual tokens from CLIP vision transformer to the early layers of LLaVA-1.5 and LLaVA-Next language models, without increasing the sequence length of visual tokens for LLMs. It doesn't require architecture change while significantly increases the number of tokens LLMs can take, so it improves accuracy especially for high-resolution images and videos.

Strengths

The authors propose an effective and simple way to increase the resolution of the visual part of VLM to increase the accuracy of the VLM, splitting the images into patches, separately applying ViT-CLIP to them, collecting (mosaic) the feature maps into a single high-resolution feature map as whole-image feature, and using residual connections to embed this feature map into the LLM. StackFormer outperforms its baseline model LlaVA on both VQAv2, GQA, POPE as well as on Text-Oriented and Zero-shot Video QA benchmarks. StackFormer achieves best performance when the backbone is fine-tuned, while when fine-tuning a backbone without a StackFormer, the improvement is limited.

Weaknesses

In Fig. 1 and 2 are missing details about the implementation of StackFormer: - how exactly you split the high-resolution image into patches - and how exactly you split the high-resolution visual tokens into different token sets with spatial dilation While you write that StackFormer achieves the best trade-off between performance and effectiveness without introducing extra visual tokens, specific indicators of overhead costs when using Stackformer are not provided. Flops, parameters, latency, memory consumption, accuracy are not provided in a single table to compare Stackformer with other VLMs.

Questions

Can you show in more detail in the Figures 1 and 2: - how exactly you split the high-resolution image into patches? - and how exactly you split the high-resolution visual tokens into different token sets with spatial dilation? Can you provide numerical indicators (flops, params, latency, memory consumption) that Stackformer achieves the best trade-off between performance and effectiveness compared to other VLMs?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

Limitation and Future Work: The paper presents limited options for processing high-resolution images by naively splitting the image into many patches and applying VIT-CLIP to them separately. While there are many approaches to processing high-resolution images using transformers: ViTDet [1], SwinV2 [2], Patch-Fusion [3], ... or simply naively resize the ViT-CLIP model to the required image resolution, using 2D interpolation of the pre-trained position embeddings [4]. 1. Exploring Plain Vision Transformer Backbones for Object Detection, Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He, 2022 2. Swin Transformer V2: Scaling Up Capacity and Resolution, Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo, 2021 3. PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation, Zhenyu Li, Shariq Farooq Bhat, Peter Wonka, 2023 4. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, 2020

Summary

This paper proposes a new visual token organization method. Specifically, it proposes to stack visual tokens instead of the commonly used stringing. Experiments show that the proposed StackFormer can improve performance on TextVQA, DocVQA, and InfoVQA.

Strengths

- The proposed method is novel, different from the commomly-used stringing method. - Experiments show that the proposed method StackFormer can significantly improve performance on some datasets, especially the traditional VQA datasets.

Weaknesses

- There are many typos in the paper. The authors need to improve their writing and polish the paper. Like Line 18 "StackFormeruses" --> "StackFormer uses"; Line 134: Multi-modal Language Models (LLMs). Sometimes it uses LMMs and sometimes MLLMs. Both are ok, but please use only one in the same paper. - There is no significant improvement for LLaVA-Next on MLLM benchmarks.

Questions

- In table 1, what do † and * mean? - Could you explain why StackFormer cannot improve LLaVA-NeXt on MLLM benchmarks?

Rating

7

Confidence

5

Soundness

4

Presentation

2

Contribution

4

Limitations

the authors have discussed the limitations in the paper. StackFormer cannot significantly improve larger model LLaVA-NeXt on MLLM benchmarks. It may be another limitation.

Summary

This paper proposes a method to add more visual information to a MM-LLM without increasing the number of tokens processed by the model. The idea is simple, just add visual tokens to the existing hidden representation between each layer of the transformer. The approach is evaluated on many tasks and shows good results.

Strengths

The paper is well written, the approach would be reproducible from the given descriptions. The idea is novel and simple and effective. The experiments are thorough.

Weaknesses

Overall the paper is well done. The experiments are thorough, the idea is well explained, and the method is reproducible.

Questions

None

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

na