AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation

Summary

This paper introduces an adaptation framework AWT to enhance vision-language models. The authors propose to augment inputs from visual and textual perspectives. The augmented inputs are dynamically reweighted based on the prediction entropy. Finally, the authors propose to use optimal transport to find the best prediction from VLM. The experimental results demonstrate strong performance under various settings.

Strengths

- This paper is well illustrated and easy to follow. - The proposed AWT framework is novel and effective, which improves performance of VLM without additional training costs. - The experiments demonstrate the strong performance of AWT under image/video tasks.

Weaknesses

- Performance-efficiency trade-off under few-shot settings. In supplementary material Figure 8, the performance increases with more augmented images under a zero-shot setting. I'd like to see a similar analysis under the few-shot image classification setting, which can further improve the significance of this work. - How do the authors decide the hyper-parameters? The authors provide the average results of 14 datasets and the OOD under different choices of hyper-parameters in Table 4. However, it seems to lack details about how the hyper-parameters were chosen. Could authors provide more information on this? - Why choose flipping as one of the image augmentation strategies? If I understand correctly, image augmentation aims to enable the VLM to capture the fine-grained matching between image regions and different text descriptions, which is already achieved by random crop.

Questions

Please see the weakness above.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation of this paper is sufficiently discussed by the authors.

Summary

The paper considers a task of low-shot (zero-shot and few-shot) adaptation of vision-language models (VLMs). The proposed approach is based on diversifying the inputs of VLMs using augmentations for images and LLMs for class names. Each input is then weighted based on its prediction entropy, while optimal transport is employed to measure the distance between the test image and classes. The evaluation is performed across a wide range of datasets for different tasks, such as zero-shot classification and video action recognition, few-shot classification, and out-of-distribution generalization.

Strengths

- One set of hyper-parameters is used across all datasets, which differs from most other methods that tune hyper-parameters per dataset, sometimes even per shot. This makes the proposed method really a low-shot one, as others tune hyper-parameters on a large annotated validation set. - The paper tackles an important problem of adapting VLMs in low-shot settings. This is a very active area of research, and the proposed method is achieving state-of-the-art results across various benchmarks. - The method is presented in a simple way that is easy to understand.

Weaknesses

- Prompts for LLM to generate class descriptions contain a description of the dataset, which is a slight limitation. It would be beneficial to show the result for prompting without knowing the dataset description. CuPL and VisDesc comparisons in Table 4d are going in this direction; however, considering that the LLM version could be different, they are not a direct comparison. - The paper shows that simple averaging of features from multiple image and text inputs does not work. However, a comparison with weighted averaging (weights based on prediction entropy) is missing. This would give a proper indication of how important optimal transport is in the method. - The related work section could be significantly extended, e.g., methods based on adapters (CLIP-Adapter [1]) and based on memory (TIP-Adapter [2], APE [3]) are not mentioned at all. Additionally, a comparison with APE for few-shot classification would be beneficial. *Minor comments:* - Are $\gamma_v = 2$ and $\gamma_t = 2$ as in L217 or are the equal to $1/2$ as in tables 4e and 4f? *References:* [1] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao, "CLIP-Adapter: Better Vision-Language Models with Feature Adapters", IJCV, 2024 [2] Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li, "Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification", ECCV, 2022 [3] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, Peng Gao, "Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement", ICCV, 2023 **After rebuttal** The rebuttal has adequately addressed all of my concerns from the initial review.

Questions

- What is the performance of the proposed method when using LLM prompts that do not include a description of the dataset? - What is the performance of the method without optimal transport but using weighted averaging of features?

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have adequately addressed the limitations of the work.

Summary

This paper proposes a novel adaptation framework for vision-language models, which replaces point-to-point alignment of text and images with set-to-set alignment. Specifically, the authors use image transformation and LLM generation to create augmentation sets. After weighting the augmentations based on entropy, optimal transport is used to compute the distance between vision and textual sets to measure their semantic correlation. Experiments are conducted on standard benchmarks.

Strengths

1. Replacing point-to-point distance measurement with set-to-set distance measurement is a significant improvement in measuring semantic correlation. 2. The proposed method is well-motivated, with its three modules—augmentation, weight, and transport—being closely related, and their effects are ablated separately. 3. Experiments are conducted on a wide range of tasks.

Weaknesses

1. PLOT [71] and Wang [73] also introduce optimal transport (OT) into vision-language models and are closely related to this work. The relationship and differences between these works need to be clearly clarified. 2. Some recent works should be acknowledged. For example, [a, b, c] have also been proposed for the adaptation of vision-language models. These works were accepted by CVPR 2024 and typically appeared on ArXiv in March, about two months before the NeurIPS submission deadline. Therefore, it is recommended to discuss these works as well. 3. The computational complexity should be analyzed and compared against prior methods. [a] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models, CVPR2024 [b] MMA: Multi-Modal Adapter for Vision-Language Models, CVPR2024 [c] Efficient Test-Time Adaptation of Vision-Language Models, CVPR2024

Questions

See weaknesses, especially the difference and connection with [71,73]

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

This paper proposes AWT, an adaptation framework that can boost pre-trained vision-language models (VLMs) for understanding new concepts on new classes. To summarize, AWT first do augmentations on both visual images and textual class names. Then an entropy metric is employed for weighting multiple augmented data. Finally, it measures the distance by OT algorithm associated with the confidence weight computed from the W step. AWT performances best in multiple scenarios including ZS, FS image classification and action recognition and OOD generalization.

Strengths

1.This paper is well-written, and well-organized with clear statements, standard structure, intuitive motivation, etc. 2.AWT shows superior performance and strong generalization ability across multiple evaluation benchmarks. 3.Utilizing LLM to augment existing class names seems interesting and effective, also distinctive from previous works. May bring new insight for the community.

Weaknesses

1.Augmenting images by simple random resized cropping and random flipping seems to be too naïve compared with nowadays prompting methods. As in previous work SuS-X [1], a much more diversified Support Set can be constructed by stable diffusion and retrieval. Meanwhile, according to Tab 4(a), we can not see any improvement by introducing visual augmentations (Raw inputs vs. A(Img.)). [1] SuS-X: Training-Free Name-Only Transfer of Vision-Language Models. Vishaal Udandarao et al. 2.The final inference step, measuring distance across two modalities by OT algorithm is not new as it was proposed in [2] and [3]. However, the difference with these two studies is not clearly claimed in the AWT paper. The original statement “Distinct from these two studies, our research diverges by eschewing the need for additional training resources, opting instead for an augmentation-based direction” looks confused and fuzzy. [2] Tuning Multi-mode Token-level Prompt Alignment across Modalities. Dongsheng Wang et al. [3] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models. Guangyi Chen et al.

Questions

1.What about applying more complex visual augmentations more than random cropping and flipping? How about introducing additional visual training data like Support Set in SuS-X? 2.Why only 50 class descriptions M are generated? The performance seems not saturated at M=50 according to Tab 4(c). 3.Any inference speed limitation? Since 50 times more images and class descriptions are generated by augmentation. 4.Are the evaluation results stable or constant? If AWT generate different image crops or different class descriptions (affected by GPT version) in multiple experimental trials, then the weighting and OT processes should be affected, which may give different final distance measure. If so, then the mean and variance of several rounds of experiments should be reported.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

No limitation discussion in the original paper.

Rebuttal reply

The rebuttal has adequately addressed all of my concerns from the initial review. As a result, I will raise my score to accept.