Visual Perception by Large Language Model's Weights

Summary

This paper presents VloRA, a paradigm for building MLLMs, which aligns visual features with the parameter space of LLMs. By representing visual information as model weights, no visual tokens is need in the input, which reduces the length of the input sequence and improves efficiency.

Strengths

(1) The motivation is convincing and the problem to solve is important. The enormous computational cost limits the training and inference devices for MLLMs. (2) VLoRA significantly reduces the FLOPs of MLLMs both in training and inference. Moreover, although not mentioned in the paper, I believe VLoRA can also reduce the consumption of GPU RAM. (3) Experimental results show that VLoRA maintains competitive performance.

Weaknesses

(1) The paper presents the FLOPs advantage of VLoRA. However, FLOPs cannot reflect the real latency of LLMs sometimes, especially when generating a long sentence. The generation of LLMs has two stages: prefilling (calculating the KV cache and generating the first token) and decoding (generating subsequent tokens one-by-one). When generating a long sentence such as image captioning, shorter inputs can significantly reduce the prefilling time, but the decoding time is primarily determined by the length of the output. (2)The experiments are not sufficiently comprehensive. To benchmark against LLaVA-1.5, the (zero-shot) comparisons should also be conducted on the following tasks: VQAv2, GQA, TextVQA, VisWiz, POPE, SEED, MM-Vet. I am particularly curious about the results of TextVQA because previous papers have shown that the performance on this dataset is strongly correlated with the number of visual tokens. (3) VLoRA can be regarded as using vision features to generate the PEFT parameters (LoRA) of LLMs. Therefore, some similar work listed as following should be discussed in detail: [1] HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks. ACL 2023 Findings. [2] LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. ICLR 2024. [3] Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning. ICML 2024.

Questions

(1) Please provide the experimental data about the real training speed and GPU RAM requirement of VLoRA and LLaVA. (2) Please provide the experimental data on the inference efficiency of VLoRA and LLaVA, especially on long-sequence generation. (3) Some recently similar work should be discussed in detail. (4) How about the results on VQAv2, GQA, TextVQA, VisWiz, POPE, SEED, MM-Vet. (5) Why use CapsFus-30m instead of blip-558k for pretraining? Can VeLoRA still be competitive when using a smaller blip-558k for pretraining?

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

This paper discusses some of the method's limitations, but I believe the discussion can be more comprehensive. See the 'Weaknesses' mentioned above.

Summary

The paper proposes a novel way to enable visual understanding in LLMs. Instead of encoding image as visual tokens, the paper proposes converting visual input to low-rank perceptual weights which are merged with LLM weights (similar to LoRA). The paper shows that the proposed approach achieves comparable performance on various single image V+L benchmarks, while significantly reducing computational cost.

Strengths

The paper presents a novel idea of encoding visual information as low-rank perceptual weights instead of visual tokens. It's a fresh perspective on how to integrate visual knowledge in LLMs which hasn't been done before. - The results show that the performance is on-par with existing methods on most standard V+L benchmarks, while requiring significantly less computational overhead (as measured by GFLOPS). - The authors show exhaustive ablations for the perceptual weights generator which were quite insightful.

Weaknesses

While the method is interesting, and novel, several practical questions remain that affect the flexibility of that method: - How will the model work when more than one image is used as input (such as interleaved image-text dialogue, videos, etc). - While the method achieves better GFLOPS than existing paradigm of using visual tokens, practical advancements (FlashAttention, KVCaching) significantly reduce actual computational overhead of adding more tokens. Can the authors comment on how their model compares after accounting for these tricks that people use to speed up inference? The authors can consider reporting tokens/s, and time to first token instead of GFLOPs. - I would have liked to see results on benchmarks which require fine-grained image understanding and spatial understanding and will potentially benefit from high-resolution like TextVQA, OKVQA, DocVQA.

Questions

- Can the authors comment on memory overhead during training? I imagine that storing the weights of perceptual weights generator (especially using different ones for different weight type) is expensive? - What is the red dotted vertical line in Figure 4 (Left)

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss some limitations, but miss many as pointed out in the weakness section. E.g., theoretical vs practical benefit when utilising modern tricks to speed up inference, as well as how to model multiple images.

Summary

The work proposes an efficient setup for incorporating non-text modalities into pretrained LLMs for reasoning-based tasks. Instead of introducing new tokens into the LLM, they propose to dynamically generate LoRA weight matrix residuals for the linear projectors within the LLM, conditioned on the input image. The weight matrix updates, when applied to the LLM, will then alter how the LLM processes the input text. Experimental results demonstrate competitive performance on QA tasks without the quadratic cost associated with increased input tokens.

Strengths

- Simple and efficient approach with effective results - Detailed analysis on architecture and the effect of the rank of the update matrix.

Weaknesses

- I think it's worthwhile to look beyond the QA benchmark numbers to understand what the implications of this architectural change are. If I am to understand this method correctly, the weight generator is not conditioned on the text-to-be-ingested by the LLM, which means it could drop information not typically useful for the task it's trained on. It would be curious to see how this approach compares to the more standard approach when it comes to asking questions about very obscure (or spatially tin) elements within an image. - The approach also appears to have a pretty sensitive sweet spot for rank, which could be expensive to tune for - From the perspective of technical novelty, I believe this is closely related to HyperNetworks https://arxiv.org/pdf/1609.09106 of which there are also transformer variants: https://arxiv.org/pdf/2106.04489 . The authors should probably include a relevant related-works section for this as well, and perhaps some additional comparisons against adaptation techniques proposed there as well. I think the goal would be to demonstrate that the proposed approach works best when it comes to sample-level adaptations, as compared to the typical task-level adaptations.

Questions

See weaknesses

Rating

5

Confidence

4

Soundness

4

Presentation

4

Contribution

2

Limitations

Yes

Thanks for the rebuttal.

Thank you for answering all my questions. New analysis to measure training and inference efficiency (requested by multiple reviewers) is much appreciated and makes the paper more thorough. It was also great to see experiments on more fine-grained benchmarks. Even though the numbers are lower than LLAVA, it shows opportunities for future work. I have increased the score to 6 (Weak Accept).