Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

Summary

This work presents MTV to enhance the in-context learning (ICL) capabilities of LMMs, which typically have limited context length, especially when dealing with multimodal data that includes both text and images. MTV addresses this by compressing many-shot examples into compact implicit representations within the model's attention heads. The approach involves calculating mean activations of attention heads across multiple inference iterations and selecting optimal locations for these activations using an adapted REINFORCE algorithm. This method allows LMMs to effectively use more examples than their context length permits, improving performance on various vision-and-language tasks without finetuning. Experiments demonstrate that MTV not only scales with more examples but also generalizes to similar out-of-domain tasks and works alongside explicit few-shot examples, offering a practical solution for efficient and effective many-shot multimodal ICL.

Strengths

1. Interesting idea and finding, MTV is an efficient and lightweight method to use training data and off-the-shelf LMMs. 2. The writing is well-written and easy-to-follow overall. 3. The results seem good, models are evaluated on two tasks, four benchmarks and three models. In addition, the efficiency is also evaluated. I like the efficiency evaluation.

Weaknesses

1. Minor: the dataset can always be included more: Try MMMU[1], OpenFlamingo and Otter did worth with 1/3/5-shot examples. 2. Minor: the models can also be included more: Try Mantis[2] [1] Yue et al., MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. 2023 ArXiv. [2] Jiang et al., Mantis: Interleaved Multi-Image Instruction Tuning, 2024 ArXiv.

Questions

1. Can this method work on LMMs that can only take in one image? Because most of models currently being evaluated are models that can take multiple images. Since your work is like to use the in-context examples as the prior, then I think it's possible.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No significant limitations

Summary

This paper should be desk rejected as the single PDF submission does not have the paper checklist. It would be unfair to make an exception as this requirement has been clearly stated at https://neurips.cc/Conferences/2024/CallForPapers and the latex template file.

Strengths

NA

Weaknesses

NA

Questions

NA

Rating

1

Confidence

5

Soundness

1

Presentation

1

Contribution

1

Limitations

NA

Summary

In the context of multimodal understanding, this work studies the area of many shot, long-context ICL (in context-learning) in natively multimodal models (large multimodal models i.e. LMMs, where images and text are interleaved). The premise proposed is that the pretrain time context of existing LMMs is prohibitive w.r.t generalization to longer contexts for adding many-shot multimodal examples. To leverage many examples under this context budget, this work proposes a technique to compress i.e. encode these multimodal exemplars into the weight space (multimodal task vectors i.e. MTV) by encoding them via mean attention head activations and replacing these mean activations into heads which are most aligned for a downstream task (which is potentially out of domain / zero shot). This selection is done at inference time via a REINFORCE based selection mechanism. Empirically, consistent improvements are demonstrated compared to the ICL baseline across open models (where weights can be accessed) on 4 tasks covering visual question answering and object classification.

Strengths

This paper addresses and important and relevant area of multimodal Q/A or understanding in natively multimodal models, where not many benchmarks exist to understand the long context multimodal capabilities of LMMs. I like the coherence in the methodology and the stepwise presentation in the paper. Figure 1 gives a concise overview of the method being proposed.

Weaknesses

While the empirical improvements are encouraging, following are some weaknesses where I have concerns regarding the motivation of the method and it would be great to have a discussion on these: 1. How is the proposed method uniquely situated in the context of "multimodal" QnA? The context length limitation, encoding exemplars efficiently and tuning head selection for a downstream task seem to be applicable in the text-only domain too. Can this same technique be applied to the text-only space as well? What is specific about the methodology proposed which would make sense for the multimodal domain but probably not work for text? 2. There are repeated claims on images being "more expensive" in the token space e.g. “Image embeddings are expensive”, “the images require more tokens to embed”. I believe these claims could be concretized better by doing a simple analysis of difference in tokens, because I believe this difference is usually not prohibitive. It would also be nice to have numbers on the context lengths in the benchmarks mentioned. 3. There is no evidence of degradation of model performance with increasing context length in the number of multimodal shots. Without this, it is hard to understand the motivation for choosing to encode shots in the weight space. If one can see the degradation being more rapid for vanilla ICL compared to this method, the motivation would be much more convincing. 3. Why does one need REINFORCE based selection? Concretely, how would the current method compare with the baseline of simply replacing all heads with the mean activations? It is unclear why the authors chose to use REINFORCE. 4. Having many shots in the context (v/s in the weight space) provides the benefit of interpretability - what is the intuition on losing on this benefit? Does shot quality in the textual space correspond to shot quality in the weight space? As a follow up, how does the "cost of prompt iteration" compare between vanilla ICL v/s MTV? 5. How robust is this technique to shot quality e.g. noisy exemplars? Is it more or less robust than vanilla ICL? 6. The empirical results focus on comparisons with vanilla ICL. While encouraging, it would be convincing if the method shows performance improvements on long context tasks where vanilla ICL fails but this method works. Further, the tasks considered are still limited and a wider range of longer-context multimodal benchmarks could be considered. (to name some e.g. MMMU, TextVQA, DocVQA).

Questions

Besides the above, here's a suggestion: there are many abbreviations (e.g. VL benchmarks) which could be expanded clearly.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Limitations are mentioned.

Summary

In-context learning with many examples can be effective for learning new tasks. However, there are challenges with many-shot multimodal in-context learning (ICL), such as the limitation caused by the model’s context length. This issue is more challenging in the multimodal setting because it processes both images and text, requiring additional tokens. Therefore, a method to compress this information into fewer tokens without finetuning is necessary. The paper proposes using large language models (LLMs) to perform many-shot in-context learning with multimodal task vectors (MTVs). MTVs are compact, implicit representations of in-context samples, compressed by the model’s attention head. The paper provides experiments showing that outperform existing methods.

Strengths

The paper is generally well-written. The motivation is clear, and the research question, and challenging and up-to-date. The paper provides experiments showing that the proposed method outperforms existing methods. The paper also provides ablation studies to better analyze and understand the model.

Weaknesses

I would recommend the authors compare their work with “Many-Shot In-Context Learning in Multimodal Foundation Models”. I think this can be a concurrent work, however, a general comparison of the methods and adding the reference to the paper helps the paper. I am a bit confused with the number of data used in each case in Tables 1 and 2. Can the authors elaborate on that? Also, can the authors explain more about the MTV+ 1-shot ICL setting? Is the setting for MTV in Table 1 2-way 1-shot always? While the FV and VTV are other existing methods, the paper does not clearly explain those in the main text and related work. I would recommend adding an extra link to the explanation on the supplementary material. Also, would the authors elaborate on the main differences here as well? Based on Figure 2, the best shot is 16 and iterations 100. However, the paper does not provide any data on shots more than 16 and iterations more than 100, which makes the conclusion suboptimal. Similarly, the paper discusses the effect of permutation only with one experiment (running for different seeds) which is not enough. Writing: In line 227, (3) is extra Referring to steps 1 and 2 in lines 42-55 while these steps are not clearly defined makes it hard to follow the text. I would recommend the authors explain more about the notion of interleaved data in the paper. The current writing can be difficult to read. In section 5.2, the reference to the table is missing Reference missing on “scaling on flowers dataset” section References to sections are missing in A1

Questions

please refer to the weaknesses.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The paper includes limitations.

High-level Summary

We thank the reviewers for their valuable feedback. In this work, we present Multimodal Task Vectors (MTV)--- compact implicit representations of many-shot in-context examples compressed in the model’s attention heads— and leverage them for many-shot in-context learning in LMMs. We are encouraged that the reviewers find our work and findings to be "important" (``dwXj``), "challenging" (``3gQC``), and "interesting" (``Xn5t``). The reviewers also noted the strength of our MTV method, which "outperforms existing methods" (``dwXj``), as well as the insightful nature of the ablation studies and efficiency results (‘Xn5t’), which help to ‘better analyze and understand the model’ (``dwXj``). Furthermore, they all noted our paper to be "well-written" (``dwXj``) and "coherent" (``3gQC``), especially appreciating the "step-by-step" (``Xn5t``) presentation and the clarity of Figure 1. We address all concerns separately in the section and look forward to an open and constructive discussion with the reviewers.

response to the rebuttal

I thank the authors for the rebuttal. After going through the answers, I stick to my current rating.