Matryoshka Query Transformer for Large Vision-Language Models

Summary

This paper introduces a new concept called Matryoshka Query Transformer (MQT) that brings in the concept of Matryoshka information packing to make visual tokens flexible and can be used in multimodal vision language models like LLaVA. The reduced tokens tend to reduce the quadratic complexity of the language model due to potential flexibility in the processed visual tokens. Owing to this, there are significant efficiency benefits all the while retaining accuracy across various benchmarks. The paper also showcases analysis and further discussion on the technique. ---------------------------- The review will be short and does not reflect the time put in for the review or the quality of the paper. When the ideas are simple and clear -- I tend to write shorter reviews to the point.

Strengths

I will go sequentially 1) The paper is extremely well-written and easy to follow. 2) The core ideas, while not completely novel as mentioned by the authors, Matryoshka and the Query transformer, making them work together and showing incredible benefits is commendable and is a worthy contribution. 3) The modelling very clear and the mechanism to achieve and the details are very well fleshed out. 4) The experiments are extensive, with a good analysis. 5) Great job with visualizations and nice to see some TFLOPs measurement. 6) Good analysis beyond benchmark numbers. Overall, this is a solid work with practical utility. I have a few questions about the paper mentioned in the weaknesses and would appreciate answers for them in the rebuttal, however, I am happy with the paper and am willing to champion it unless there is something I am missing found by other reviewers.

Weaknesses

Most of these are not weaknesses, but rather questions 1) How are you measuring TFLOPs? What does it include, the cost of vision encoder? LLM processing and generation? I might have missed this in the paper and any pointer would be great. 2) While I understand the when you are picking a fixed # token for MQT, you make that choice before in hand. What happens if you actually just obtain 256 tokens and take the first 8? What will be the performance in that case? I know softmax etc make huge differences, but would be good to see that. 3) Any thoughts on sampling vs joint training? 4) Why is log-based (actually exponential) spacing so poor (not 3.5% as mentioned in the paper but 2.1% from the table) so much worse than fine granularity? This is a bit surprising to me because 256 token performance should not be affected because of the lower token counts. 5) In visualizations, what do you mean by one random token? I am looking forward to your answers on these things.

Questions

See above.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Summary

Authors try to use Matryoshka mechanism to guide the learning process of LVLMs such as llava. Authors show a pretty good scaling curve with different amount of visual tokens.

Strengths

1. The idea is interesting. 2. The presentation is clear. 3. The attention visualization is interesting.

Weaknesses

1. For a given image, which scale should I choose to achieve best trade off? Is there an answer? 2. Authors use more compute (2 epoch) to get the similar performance as LLaVA, is there any reason for this? 3. All designs are center upon the qformer. There is little study on this. How many layers do we need? Do we really need it?

Questions

1. What are the performance under other datasets like TextVQA and ChartQA? Those are dataset different from the natrual image distribution. 2. The visualizations are good but we don't have quantitive numbers to support your claim. Since those pictures can be cherry picked.

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

See comments above.

Summary

The paper considers multimodal vision transformers, in which we have a stream of both visual and textual tokens. Current architecture typically assume a fixed number $m$ of visual tokens. In contrast, the proposed `MQT` aims to achieves a dynamic number of visual tokens. This is done by integrating a form of compression+dropout mechanism during training. Specifically, the image tokens are compressed to $M$ tokens using cross-attention. To vary the number of tokens, the model only needs to use only $m < M$ queries from te cross-attention mechanism. At training time, this is done by randomly selecting $m$ and feeding the **first** $m$ queries.

Strengths

* The proposed method is a simple idea that can be added as plug-and-play to any model * generally the idea of using some form of dropout also often has benefits for regularization so it may also benefits more than efficiency * The paper is well written and has comprehensive ablation experiments

Weaknesses

- **Lack of comparison with dynamic baselines**: in the field of image-only ViT, there is a flourishing literature on dynamically reducing the number of tokens using merging or pruning or scale selection. I am not as familiar with this literature for multimodal models, but there does seem to be similar existing approaches: for instance, *LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models* is one, though it might be too recent for the authors to have considered it for the submission. However, more generally, having a simple off-the-shelf token merging/pruning approach for the ViT encoder would be interesting. - **No improvement over baselines in the low FLOPs regime**. From Figure 1, it seems that the benefits of the proposed `MQT` decrease in the low FLOPs regime, when increasing the sparsity ratio of $M / m$. This limits the usefulness of the methods in the realm of model efficiency, though it seems to be a useful and simple training strategy for the high FLOPs regime.

Questions

- How are the $m$ latent queries chosen at inference ? I am assuming $m$ is chosen as a hyperparameter, and the first $m$ queries are kept, however it is not very clear from the methods section - Addressing the **low FLOPs regime**: From Figure 1, it seems that the benefits of the proposed `MQT` decrease in the low FLOPs regime, when increasing the sparsity ratio of $M / m$. I think it is not necessarily a surprising result as increasing sparsity too much will severely impact accuracy. However, I am wondering how would the trade-off curves would look like when starting with a base `QRT` model with a smaller $M$ and lower sparsity ratio, i.e. with a model that would achieves same number of tokens/FLOPs, but where the relative sparsity ratio to achieve is less aggressive.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper has a limitation section but in my opinion it does not fully address some limitations of the paper (e.g. no improvement in performance in the low FLOPs regime)

Summary

This paper addresses the challenge of achieving flexibility in the number of visual tokens to suit different tasks and computational resources. Inspired by Matryoshka Representation Learning (MRL), the authors propose the Matryoshka Query Transformer (MQT), which allows for any number of visual tokens during inference. Experimental results demonstrate considerable performance across varied visual token lengths and show promising results even with extreme token numbers.

Strengths

1. The problem of handling an arbitrary number of visual tokens is highly valuable. The authors propose a novel MRL-based method to tackle this issue. 2. The experimental results are promising, highlighting the trade-off between performance and computational resources. 3. The analysis of the impact of visual tokens is valuable, providing meaningful conclusions that deepen our understanding.

Weaknesses

1. Can you explain the choice of the MQT transformer design? Does it need to maintain sufficient capability for extracting information from varied tokens? For example, Q-Former has multiple layers to achieve effective information extraction. 2. The extraction process V=Q(Z,G) lacks correlation with textual information. In Fig 4, different text prompts should result in the same attention map due to this. Therefore, this design might be limited when dealing with complex images, where visual representations should maintain different semantics. 3. How does the model perform in text-rich scenarios, such as TextVQA? The sophisticated recognition ability might be significantly compromised in these cases. 4. Since the number of visual tokens affects performance, how does the model perform when the number of visual tokens exceeds the original 576 tokens? 5. It would be better to provide a comparison with the LLaVA baseline, using adaptive pooling or vanilla Q-former to adjust token numbers.

Questions

As in LLaVA-1.5, the 576 visual tokens do not introduce significant computational overhead. It would be more meaningful to evaluate this method on a more extensive case like LLaVA-NEXT.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Please consider answer the questions above.

Reply to authors

Thank you authors for providing the rebuttal. I am still doubtful about the visualizations and even overall the story line provided. The supervision signal under different granularities of visual tokens are from the same set of captions. How can we make sure this makes sense and model really learns the correct granularity? If so, then the assumption about the visualizations do not make sense.