Meta-Adapter: An Online Few-shot Learner for Vision-Language Model

Summary

For the Vision-Language Model task, which usually requires a small number of samples for fine-tuning, this work proposes a Meta-Adapter method. The Meta-Adapter method is based on the gated multi-head attention mechanism, and can be generalized to unseen categories without additional fine-tuning after a small amount of training by Few-Shot. The effectiveness of this work is experimentally demonstrated.

Strengths

1. From the experimental results, this work performs better in corss-dataset generalization and has good performance on downstream tasks. 2. This work is well-written, the related works are well summarized and the contributions are clearly demonstrated. The empirical performance is promising.

Weaknesses

1. The explanation of the formulas in this work could be more clear to help the reader understand the content. 2. The connection between the meta-learning method and Meta-Adapter is not well explained.

Questions

1. Why is it called Meta-Adapter? In my understanding, the method of this work is to generalize to new samples by using a small number of training samples, and this method is not called meta-learning. 2. The motivation for this work is that "few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters on few-shot samples ", and the same motivation for Tip-Adapter. It seems to me that Meta-Adapter is an improved approach based on Tip-Adapter, is there a new improvement in the motivation for the approach to this work?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes a meta-adapter structure for CLIP like vision-language backbone. Specifically, a cross-attention with a gate mechanism are used to construct the meta-adapter. It aims to improve the few-show learning ability for current CLIP backbone. Compared with other baselines such as CLIP-adapter and Tip-adapter, the proposed meta-adapter obtains improvements on several experimental settings.

Strengths

1. This paper studies a popular question about CLIP few-shot learning. 2. The motivation and idea are well presented. 3. Meta-adapter is straightforward and easy to follow.

Weaknesses

1. The meta-adapter uses a cross-attention and gate module, which are commonly used for this field resulting in the limited technical contribution. Simply adding these modules to improve the final performance is not very surprising. 2. The proposed module is general for vision-language model, but the evaluation is only for classification. Supplementing retrieval evaluation will further support this method. 3. Some figures are relatively misleading the readers. For example, in figure1, please unify the number scales in the performance axis. Current version is a little confusing to show the performance gain. 4. As shown in Tab1 and the limitation section discussed by the authors, the general performance gain is not very significant.

Questions

Please check the weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Discussed by the authors.

Summary

The main goal of the paper is to explore an approach that is light-weight to allow a CLIP-pretained model to perform well in few-shot settings. The proposed approach (called Meta-Adapter) essentially learns an additional multi-head attention network with an additional gating function. The approach is simple and seems to achieve interesting results.

Strengths

The addressed problem is of great practical relevance and pretraining with CLIP is highly popular these days Overall the idea is quite simple (which is positive) and intuitive The experiments give a variety of comparisons to zero-shot CLIP and TIP-Adapter

Weaknesses

The paper mentions on line 61 that the paper performs abalation studies - but honestly the reported experiments in the main paper are not real proper ablations - in particular the paper should have ablated the design choices of the approach where I could not find any experiment - since the approach is so simple that would have been very easy to do and would have helped to understand the approach better The paper reports slightly lower "base class" performance (e.g. table) 2 for most datasets - which is then "compensated" for by higher "novel class" performance. This is a classic trade-off many works have - but I would have liked to not just see a single pair of base/novel class performance but rather a curve/set of novel/base class performances - that would be again more interesting and telling

Questions

see weaknesses section above The authors argue that the work is doing some sort of "meta-learning" (lines 116-118) - however, section 3 (method) does not talk about any meta-learning setting as far as I can tell - can you please give an explanation what the meta-learning part is here?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

ok for me

Summary

This paper proposes Meta-Adapter which can refine the CLIP features guided by the few-shot samples in an online manner. The major challenge of adapting CLIP with few-shot samples is over-fitting. Compared with offline approaches CoOp or online approaches TIP-Adapter, Meta-Adapter alleviates the over-fitting problem and demonstrates superior generalization across datasets. The author further adapts the Meta-Adapter to an open-vocabulary object detector, ViLD, and also finds decent improvements

Strengths

+ This paper is generally well-written and easy to follow. + The proposed Meta-Adapter shows improved generality over different datasets compared with the Tip-adapter baseline. + Meta-Adapter can be served as a plugin module for open-vocabulary models, such as ViLD.

Weaknesses

- Limited comparisons with existing approaches. Even though there are few online approaches like Tip-adapter, it is also encouraged to include more competitors, such as offline approaches in the experimental discussions. - The authors claim that Meta-Adapter uses 'a lightweight residual style adapter'. However, what is the optimal size of the adapter is not justified in ablation.

Questions

- The captions of each figure are short. I recommend adding more detailed explanations for each figure.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes an online adaptation method for CLIP (no fine-tuning of few-shot samples of unseen categories are required, unlike CoOp, CoCoOp, and CLIP-Adapter), called Meta-Adapter. The main claim seems to be that Meta-Adapter is more robust than the most related approach Tip-Adapter, which relies heavily on the hyperparameter search strategy on the target dataset – alpha and beta in Eq (2) which help adjust the weight between category embeddings and few-shot visual embeddings. More specially, the high-level idea of Meta-Adapter can be found in Figure 2 and Eq. (3). One can make a comparison between Eq. (2) and Eq. (3) to see how Meta-Adapter simplifies the online adaptation. The architecture of Meta-Adapter is based on the gated multi-head attention mechanism [26].

Strengths

S1: Even though the design choice for Meta-Adapter seems arbitrary, it is simple and sound. S2: Experimental results show the superiority of Meta-Adapter to Tip-Adapter across tasks/settings and architectures (Table 1-5).

Weaknesses

W1: Experimental settings seem to deviate from those in Tip-Adapter, making them less convincing. Examples: (i) This paper considers 8 image classification datasets instead of 11. (ii) The efficiency analysis is lacking. (iii) Related to (ii), there are no CoOp, CoCoOp, or CLIP-Adapter baselines. (I understand that these are online approaches but it would be nice to discuss, for example, an effectiveness/efficiency tradeoff). W2: Clarity. This point is minor but overall the paper seems to jump into details too quickly and it would benefit from more coarse-to-fine writing. For example, Section 3.2 could benefit form describing different components before jumping into details.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Good rebuttal, please keep additional experiments in the paper

Thank the authors for the detailed rebuttal. It resolved all my concerns and I highly encourage the authors to include the additional experiments in the main paper or appendix. I have raised my score to 5: Borderline accept.

Post rebuttal

Thank for the authors' feedback. Even if I agree this draft focuses on a new framework instead of structure, I still concern the proposed framework is more like a technical combination. Thank you for reminding the other experimental setting other than classification, actually, my concerns was more about if the retrieval task can be added. I also checked other reviews. My concerns have been partially resolved and I hope the authors can revise/supplement this draft accordingly. I raise my score to 5. Thanks!

Post rebuttal

I am very grateful to the author for his efforts in the rebuttal process. The author's rebuttal partly resolved my doubts, and I decided to maintain the previous rating unchanged.