LIVE: Learnable In-Context Vector for Visual Question Answering

Summary

This paper aims to improving the in-context learning performance of multimodal models on VQA tasks. The paper proposes a Learnable In-Context Vector (L-ICV) method to distill essential task information from demonstrations into a single vector, reducing computational costs and enhancing accuracy in VQA tasks. Experiments show that L-ICV can achieve better ICL performance compared to random 32-shot demonstrations and significantly reduce the computation cost.

Strengths

The topic is important for the community, and the proposed method is an insightful exploration that can be beneficial. The overall writing of this paper is clear and easy to follow. L-ICV reduces computational costs, requiring only a fraction of the FLOPs and inference time compared to traditional ICL methods.

Weaknesses

One of the major weaknesses is that the evaluation of L-ICV is limited to a small set of models and datasets. Only one model (IDEFICS-9B) and two datasets (VQAv2, OKVQA) are reported across the paper. This limitation raises questions about the generalizability of the method to other multimodal models and different dataset configurations. Besides, recent literature has discussed the in-context limitation of these datasets [1, 2, 6, 7] and a broader comparison of tasks that rely more on visual in-context learning ability should be beneficial to showcase the advantage of L-ICV. The second major issue is the relatively limited increase of the ICL performance. Although the L-ICV method demonstrates improved efficiency and performance, the reported increase in accuracy is not particularly significant. while L-ICV has been trained on a large dataset, the expected performance gains are not as high as one might anticipate given the extensive training resources utilized. Additionally, the paper only compares L-ICV with ICL using randomly chosen demos and does not compare L-ICV with the classical RICE (Retrieval-based In-Context Examples) method [3, 4, 5] or a recently proposed MMICES [6]. RICES and MMICES samples can lead to a direct ICL performance boost, and a direct comparison would help to position L-ICV within the current state-of-the-art. Besides, if L-ICV is trained using RICES samples rather than randomly sampled demos, will the performance be better? This should be an interesting exploration that should be conducted. Furthermore, the in-context learning ability of large models is practical and useful as it does not need any parameter optimizations and only requires a handful of demonstrations to quickly adapt to new tasks. To enhance this ability, it would be better if the proposed method also requires minimal dataset, tuning, and optimization effort to achieve similar or better performance. Overall, while L-ICV presents a valuable exploration of the field, addressing these weaknesses through broader evaluations, detailed statistical analyses, comprehensive comparisons, and further optimization efforts would enhance the method's credibility and impact. [1] Zong, Yongshuo, Ondrej Bohdal, and Timothy Hospedales. "VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning." *arXiv preprint arXiv:2403.13164* (2024). [2] Doveh S, Perek S, Mirza M J, et al. Towards multimodal in-context learning for vision & language models[J]. arXiv preprint arXiv:2403.12736, 2024. [3] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." *Advances in neural information processing systems* 35 (2022): 23716-23736. [4] Awadalla, et al. "Openflamingo: An open-source framework for training large autoregressive vision-language models." *arXiv preprint arXiv:2308.01390* (2023). [5] Yang, et al. "An empirical study of gpt-3 for few-shot knowledge-based vqa." *Proceedings of the AAAI conference on artificial intelligence*. Vol. 36. No. 3. 2022. [6] Chen, et al. "Understanding and Improving In-Context Learning on Vision-language Models." *arXiv preprint arXiv:2311.18021* 1.2 (2023). [7] Baldassini, Folco Bertini, et al. "What Makes Multimodal In-Context Learning Work?." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

Questions

Please see the Weaknesses section.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitations of this paper are threefold. 1. The experiments are relatively limited regarding the model types and evaluation tasks. 2. The proposed method requires extra training, optimization, and hyperparameter tuning effort. This added complexity could be a barrier to adoption, especially when there is only a limited set of demo data. 3. The increased performance is not significant, especially considering the training and tuning effort. This raises questions about the practical benefits of the proposed method relative to the effort involved in its implementation.

Summary

This paper proposes a method by introducing the idea of In-Context Vector in the field of NLP, by introducing L-ICV with a smaller number of parameters in LMM to replace in-context demonstrations to introduce external knowledge, and obtain through training The appropriate shift direction is combined in the query. By comparing with some existing methods, the author proves that L-ICV can significantly reduce computational costs while avoiding the bias in the selection of in-context demonstrations and improving the accuracy of VQA tasks.

Strengths

(1) The paper is clearly written and easy to comprehend. (2) The motivation for the proposed method is reasonable. Existing ICL methods are difficult to achieve good results while saving computing resources. Using a shifting vector instead of in-context demonstrations can reduce the computational cost while ensuring the effect.

Weaknesses

(1) In the comparison of Section 4.2.1, only the inference time consumption of L-ICV and k-shot ICL is listed. How about other methods like TV, FV, etc.? (2) The method's generalizability is questionable as it has only been tested on the IDEFICS model, with no consideration of other models. (3) The paper only proves the effect of L-ICV on VQA tasks, which cannot be extended to other task scenarios and has limited application scope. (4) In the ablation study, there is a lack of a control group using untrained L-ICV to prove the effectiveness of shift direction.

Questions

(1) Why 32-shot demonstrations are chosen in L-ICV training? According to Table 3. As the Number of Demonstrations k increases, the performance gap between L-ICV and ICL gradually narrows. When k continues to increase, will the performance of ICL exceed that of L-ICV? (2) Why is there a need for a unique shift effect for each layer in the L-ICV method, but in the comparative experiment of LoRA, only the LoRA module is added to the token classification head of the last layer? (3) In the training stage of L-ICV, the k demonstrations used for training are also randomly selected. How to prove that this can eliminate one of the major challenges when applying ICL: the performance is sensitive to the selection of ICDs.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations in the PaperChecklist.

Summary

The study introduces a Learnable In-Context Vector (L-ICV) for Visual Question Answering (VQA) tasks. This approach extracts task information from demonstrations to improve the performance of Large Multimodal Models (LMMs) while reducing computational costs.

Strengths

* The writing of this paper is excellent, the text is fluent, and the motivation is easy to understand. * The proposed method is novel and effective, significantly reducing the cost of reasoning. * The experiment is very solid, and the statistics in Figure 5 are impressive, intuitively showing the offset of the in-context vector.

Weaknesses

please refer to the questions.

Questions

* Is it worthwhile to increase the training cost of the extra in-context vector for the sake of reduced inference cost? I am curious about the trade-off between reduced inference time and additional training time. * How much would performance drop if each layer shared the same trained bias vector?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper introduces learnable in-context vector (L-ICV) that extends the standard ICV method by training it on task samples. The method achieves performance benefits similar to ICV and the training methodology enables the L-ICV to outperform while still preserving the benefits of few-shot in-context learning. This is achieved through the use of a KL-loss term, that preserves the behavior rather close to ICV, but allowing adaptation as necessary. The evaluation of the method is done on OKVQA and VQAv2 datasets. For both of datasets the performance of L-ICV outperforms that of LoRA fine-tuning. Authors provide an explanation and namely that due to need to access external knowledge, preserving the in-context learning behavior is advantageous compared to performing LoRA fine-tuning that disrupts the behavior, particularly when using few samples.

Strengths

I believe the authors present a well-grounded method extending the strengths of in-context learning through few-shot learning. I find the learning approach balances well the specific model behavior with the task needs. This is achieved through 1) Strong theoretical grounding and explanation of the method, contrasting well the formulation and empirical results with other standard methods, such as ICV and LoRA. 2) In-depth visualizations, qualitative assessment of the differences between the methods, such as the t-SNE visualization, decoding of the top-10 tokens and performance graphs compared effect of varying numbers of samples on L-ICV and LoRA

Weaknesses

There are a few weaknesses, when contrasting it in-depth with methods focused on ICL for visual-language models. 1) Ignoring some of the recent work on interpretation of in-context learning abilities (whether few-shot or zero-shot). It has been discussed as to whether the learning process performs task recognition (superficial format recognition, for example) or task learning (learning to map inputs to outputs) [[1]](https://arxiv.org/abs/2305.09731). The concept was extended to VLMs, by extending it to visual task recognition, studied particularly on VQA [[2]](https://arxiv.org/abs/2312.01571). It seems that VLMs suffer from image specific when doing ICL, such as short-cut inference (ignoring the image similarity, picking on similar question-answers) and others. 2) The performance of LoRA is intriguing. Although I understand the motivation to completely treat it as baseline and naively apply on the same amount of trainable parameters, I find the comparison forcefully made to fit the story of the paper. The choice of tasks that require external knowledge make it obvious that LoRA can't be a good fit. Here it would be fair to either add another task with knowledge contained or alternatively modify on what LoRA is applied (e.g. synthetic 32-shot ICL examples) to contrast on fair grounds with L-ICV. [1] https://arxiv.org/abs/2305.09731 [2] https://arxiv.org/abs/2312.01571

Questions

Questions directly map to weaknesses discussed at previous point. 1) Could authors ground the work in this line of interpretation of in-context learning and explain how L-ICV improves upon task recognition or task learning angles? 2) Could the authors extend their analysis to such a task (e.g. OcrVQA, DocVQA, ChartQA, or InfoVQA)? Alternatively, why is comparing naively LoRA with L-ICV tuned to the output behavior of few-shot in-context demonstrations a fair comparison at all? I'm willing to raise my score if the authors help me better understand these angles.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Two more questions

Thank you for the feedback. I understand that the main goal of this paper is to accelerate inference time. I agree that learning ICVs and retrieving ICDs based on a query can be orthogonal. I also appreciate the newly added experimental results and findings during training. I have two further questions below. In a realistic setting, where a new task has only a few demonstrations and no large annotated dataset for training, how can LICV be better utilized? I believe this is more practical compared to VQAv2 with 8000 samples in applications. Also, how many images are used as training data for the VL-ICL Clevr and Textocr tasks? Additionally, I am curious about the transferability of the context vector trained on one task's data to another relatively similar dataset, such as being trained on VQAv2 but tested on OKVQA. Do you have the experimental results and analysis regarding this matter?