Q-VLM: Post-training Quantization for Large Vision-Language Models

Summary

The authors present a novel post-training quantization framework for large multimodal language models to enhance inference speed. This method accounts for cross-layer dependencies that significantly impact overall model discretization errors and leverages activation entropy to effectively balance these errors with search costs. Extensive experimental results demonstrate that this framework can reduce memory usage and improve generation speed by 1.44x on the 13B LLaVA model, while maintaining comparable performance across diverse multimodal downstream tasks.

Strengths

1. The paper considers cross-layer dependencies during the quantization process for the first time, which is novel. 2. The proposed quantization method is simple and effective. It significantly reduces memory usage and improves inference speed, which is crucial for MLLM deployment. 3. The paper is well-written and easy to follow.

Weaknesses

1. Is there any quantization for the adapter between the vision encoder and the LLM? With various types of adapters such as MLP for LLaVA, Q-former for BLIP-2 [1], and Visual Abstractor for mPLUG-Owl [2], does the proposed method apply to models with different adapters? 2. Instead of comparing with outdated methods such as Q-LoRA and AWQ, it would be beneficial to compare with more advanced techniques like ZeroQuant-FP [3] and include a more in-depth discussion. [1] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." International conference on machine learning. PMLR, 2023. [2] Ye, Qinghao, et al. "mplug-owl: Modularization empowers large language models with multimodality." arXiv preprint arXiv:2304.14178 (2023). [3] Wu, Xiaoxia, Zhewei Yao, and Yuxiong He. "Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats." arXiv preprint arXiv:2307.09782 (2023).

Questions

1. Does this quantization method only work for LLaVA-like architectures? How does it apply to cross-attention based VLMs such as Flamingo [1] and Otter [2]? [1] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems 35 (2022): 23716-23736. [2] Li, Bo, et al. "Mimic-it: Multi-modal in-context instruction tuning." arXiv preprint arXiv:2306.05425 (2023).

Rating

5

Confidence

2

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have addressed the limitation in the manuscript.

Summary

This paper proposes Q-VLM, a post-training quantization framework for Large Vision-Language Models (LVLMs). It aims to reduce the model complexity of LVLMs for practical deployment by replacing float numbers with quantized ones and substituting multiply-accumulate operations with integer arithmetic. The key innovation lies in mining cross-layer dependency to efficiently search for optimal rounding functions that minimize quantization noise across the entire model. The authors also optimize the visual encoder to further reduce search costs and maintain quantization accuracy. Experimental results on LLaVA and MoE-LLaVA models demonstrate significant memory compression and speed increases without severe performance degradation.

Strengths

1. **Novel Approach to Quantization**: The paper introduces a novel approach to post-training quantization that considers cross-layer dependencies, which is a significant departure from traditional layer-wise methods. This approach has the potential to improve the efficiency and accuracy of quantized LVLMs. 2. **Clear Presentation**: The paper is well-organized and clearly written, making it easy to follow the authors' thought process and understand their contributions.

Weaknesses

1. **Limited Evaluation on Diverse Datasets**: The majority of experiments are conducted on the ScienceQA dataset, which may not fully represent the diverse range of tasks and challenges that LVLMs encounter in real-world applications. Evaluating the method on a wider range of datasets would provide a more comprehensive assessment of its effectiveness and generalizability. 2. **Marginal Performance Improvement**: The observed improvements in some cases are relatively small, potentially due to the specific characteristics of the ScienceQA task. It would be beneficial to investigate whether the method's impact is consistent across different tasks and datasets, or if its effectiveness is limited to specific scenarios. Some typo: In the abstract and introduction, the phrase "compresses the memory by 2.78x and increase the generate speed by 1.44x **about** 13B LLaVA model" may be unclear. It could be revised to "compresses the memory by 2.78x and increases the generation speed by 1.44x **for** the 13B LLaVA model" for improved clarity.

Questions

1. Can the authors provide more experiments on other tasks/datasets to demonstrate the generalizability of their method beyond ScienceQA?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

Not found.

Summary

The authors propose a new post-training quantization method for LVLMs. The authors separate several layers in a LVLM into blocks and search for the optimal quantization bitwidth for each block individually. The authors also introduced a new objective function for quantizing the vision encoder. The extensive experimental results show the proposed methods outperform the SOTA post-training quantization methods on LLaVA models.

Strengths

- The authors provide a relatively comprehensive related work discussion in the paper. - The paper is well-motivated. Post-training quantization on LLM / VLM is critical for deploying these models due to their extensive computation consumption during training. - The authors provide a large amount of evaluations and ablations in the experimental section to help other researchers better understand their model performance.

Weaknesses

- The overall methods part is difficult to follow. It is not clear if the authors adopt different quantization strategies in visual encoders and language model and the projects in LLaVA. Also, it is not clear to me why the authors design those specific methods for LVLM, i.e., why the proposed methods are particularly suitable for LVLM compared to LLM, CNN, etc. It would be better that the authors can discuss more details about their insight or motivation for those designs. - The main advantages of post-training quantization compared to quantization-aware training is that it does not need the entire training data and requires less computation. Therefore, it would be better that the authors can provide some ablations in terms of the amount of training data used during the post-training quantization and the number of computation needed during quantization. Otherwise, it is difficult to position this paper w.r.t. other relevant work. - The authors claimed the prior quantization methods are suboptimal. However, adopting entropy as proxy and separating layers into blocks are also sub-optimal solutions. Then, there exists another question to be answered, why the sub-optimal solutions proposed in the paper are better than the others?

Questions

See weaknesses

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Yes

Summary

The paper introduces Q-VLM, a PTQ framework for VLMs that leverages entropy as a proxy to manage cross-layer dependencies for both language model and visual encoder. Experimental results on ScienceQA, VizWiz, VQAv2 datasets and LLaVA variant architectures validate the efficacy of the proposed method.

Strengths

* Q-VLM achieves good performance with W6A6 on ScienceQA, VizWiz, VQAv2 compared to the FP counterparts. * Ablation studies in Table 1 and Section 4.2 are helpful for understanding the importance of each component of Q-VLM, such as cross-layer dependency mining and visual encoder optimization. * Experiments across LLaVa model sizes and bit-width demonstrates the robustness of the proposed method.

Weaknesses

There are several concerns: * The benefit and motivation for using entropy as a proxy are unclear. Why not directly use the quantization error as a metric for performing block-wise searches to mine cross-layer dependency? Are there any accuracy or inference efficiency gains by using entropy as a proxy? * The idea of mining cross-layer dependency is not new. Existing works, such as BRECQ, already address block-wise quantization. * Important baseline methods are missing in the comparison results. For a more solid comparison, include other state-of-the-art PTQ methods for the vision branch and more recent works such as SmoothQuant and ZeroQuant variants for the language branch. Comparing with the previous best method on the language model and another prior method on the vision model would provide a clearer picture of the overall improvement. * The proposed method appears to be a general approach for both language and vision models. It would be interesting to demonstrate its individual effect on either the language or vision model.

Questions

Suggest to clarify the concerns in the weakness section

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

yes

Response to Rebuttal

Prior to the rebuttal, my primary concern was the universal effectiveness of the proposed Q-VLM methodology. I am pleased to see the additional experimental results across various tasks, datasets, and models, all of which support the superior performance of Q-VLM. As such, I am raising my score from 4 to 6.