Tuning Multi-mode Token-level Prompt Alignment across Modalities

Summary

The paper aims at improving few-shot classification based on CLIP-ViT-B/16 using prompt tuning. The authors introduced a multi-modal prompt tuning framework with token-level alignment and distribution matching. The prompts are tuned on ImageNet and then the model is benchmarked on 15 image datasets.

Strengths

+ The method achieves better results than baselines (e.g., CoOp, MaPLe) on a series of datasets, including both few-shot classification and transfer learning tasks. + The code is attached, making the method reproducible. + The introduced approach should be practical in real-world applications.

Weaknesses

- The writing appears rushed and some sections are not easy to follow. There are also some typos and grammar issues, e.g., L135-136 "...CLIP. maximuming...". - The central insight of this work is to introduce token-level alignment in a multimodal prompt-tuning framework. However, the fine-grained alignment with optimal transportation is not new (e.g., Hierarchical Optimal Transport for Multimodal Distribution Alignment, NeurIPS19) and the multimodal prompt-tuning framework is similar to existing works like MPT. It provides limited new knowledge to the community. - In the experimental part, there are only comparisons based on base-size models. However, it is pretty essential to verify the scalability of the introduced method on larger models.

Questions

Please see the Weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have discussed their limitations on the GPU memory during test. It may also be important to discuss their training cost.

Summary

This work aims to overcome the limitations of previous works in prompt tuning for vision-language models. Unlike prior approaches that focus on single modality or holistic prompt alignment, the paper proposes a multi-mode token-level tuning framework that leverages optimal transportation to align prompt tokens across different modalities. The framework introduced in the paper relies on two crucial elements: multi-mode prompt discovery and token-level alignment. By enabling diverse semantic representations, multi-mode prompt discovery ensures a broader range of prompts. On the other hand, token-level alignment allows for a more detailed exploration of similarity between modalities. Extensive experiments demonstrate the effectiveness of the new method ALIGN.

Strengths

1. Overall, this work is well-motivated, and the paper is well-written. It focuses on a practical problem in prompt learning and proposes a novel method called ALIGN. 2. This work first leverages Optimal Transportation to address the limitations of prompt tuning. Intuitively, the new method can succeed in finding out token level vision-language prompts. Compared with the single-mode or holistic level prompt tuning approaches, this method can better reveal the connection between visual and textual prompts, which would be a very valuable insight for the study of vision-language alignment. 3. The authors conduct extensive experiments to evaluate their method, and the results demonstrate non-trivial improvements over the existing baseline models. The comparison is clear, and their method ALIGN achieves SoTA performance in most scenarios.

Weaknesses

1. The paper lacks discussion of computing cost. It seems that your ALIGN requires more self-attention computation compared to the baselines such as VPT and TPT. Your improvements might be challenged if the difference in training/inference cost is not provided or is too big. 2. While the paper demonstrates its merits in token-level prompt learning, more empirical results in fine-grained tasks such as semantic segmentation should be included. However, the paper only gives classification results. I understand that implementing prompts into segmentation tasks is not easy and it merely appears in your baselines, but it would be a very strong support to your ALIGN’s effectiveness and significance. 3. Some expressions are confusing. For example, in line 249-250, there is “For each task, we optimize the number of epochs”. Do you mean “the same number of epochs”?

Questions

1. How about the comparison of computing cost between your method and the baselines? 2. Is there any experimental results in inference tasks other than classification?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See "Weaknesses".

Summary

This paper introduces a multi-mode token-level alignment framework for multi-modal prompt tuning, which improves the representation of visual and textual modalities and can be used to improve existing methods. The task is formulated as a distribution matching problem, addressed using prompt and token-level optimal transportation (OT), providing a principled and elegant solution. The method is applied to few-shot classification, dataset transfer learning, and domain generalization, showing superior results on widely used datasets.

Strengths

• The learning of multi-modal, multi-mode prompts is facilitated by establishing optimal transport (OT) at the prompt and token level. • The structure of the manuscript is solid and it's well-written overall. • The efficiency of the proposed ALIGN method for both few-shot classification and generalization has been confirmed through a series of diverse experiments.

Weaknesses

• The proposed model might be memory-intensive, however, an analysis of the additional time and memory costs has not been provided. • The omission of specific details, particularly regarding the ablation analysis, somewhat undermines the impressive results. - The study does not examine the influence of prompt length and quantity on the experimental outcomes. - It remains unclear whether token-level alignment provides any enhancements compared to prompt-level alignment. • Miscellaneous issue - Figure 1 is not mentioned in the body text. - It seems that line 135 is missing a period, and the expression "maximuming" appears to be awkward. - In Table 1, CoOp shows the best result in 'Stanford Cars'-Base, so it should be highlighted instead of ALIGN

Questions

See the comments in weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

• As pointed out in the conclusion section, this paper's method still demands substantial GPU memory. • The method suggested does not appear to be well-suited for a fully zero-shot scenario where there is no training samples. This scenario is little bit different from the Base-to-New Generalization scenario.

Summary

The paper proposes a prompt-learning method for adapting CLIP to few-shot classification. The proposed prompt is multimodal, i.e., both vision and language encoder is adaptable, and multimode, i.e., each modality is assigned with several prompts for diverse representation. Experiments are conducted on few-shot and base-to-new transfer settings, showing the method's superiority over previous state-of-the-art.

Strengths

* Promising results are achieved, both in the few-shot setting and base-to-new transfer setting. * The presentation is mostly clear and easy to understand.

Weaknesses

* Incremental novelty of the paper. Visual-language prompt learning with multimode prompts and optimal transport (OT) is explored in PLOT [17]. Also, multimodal prompts are explored in works like [27][28]. This proposed multimode multimodal prompts with OT multimode seems to combine the two types of previous methods and reveals limited insights. * Missing comparison on computation complexity. As multimode prompts require quite large parameters and computations compared to a single prompt, I want to see the comparison of parameter efficiency and time efficiency. * It is strange that there is no ablation study of each component of the models. It is important to see the individual contribution of the multi-mode prompts and the token-level alignment.

Questions

* In Fig. 3, ALIGN performance on UCF is much superior to previous methods, but the superiority on UCF is not obvious in the Base-to-New setting, especially on the base classes. Why is that case? * In Page 3 Line 97, "Empirical findings" needs to specify the source of information or citing papers.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper argues that existing prompt turning fails to caption the sample diversity, learning to sub-optimal prompt discovery. To this end, they propose a multi-mode token-level alignment for multi-modal prompt tuning. Specifically, they formulate the prompt turning as the hierarchical optimal transportation problem (distribution matching problem). As a result, the extensive experiments on few-shot image classification, transfer learning, and domain generalization, show the superiority of the ALIGN.

Strengths

1. **The proposed method is effective and comprehensive**. Learning and aligning a set of prompt tokens across modalities by hierarchical optimal transportation is effective and simple. 2. **The result is strong and the evaluation is comprehensive**. The extensive experiments on 15 widely used image datasets under the setting of four task settings show the superiority of the proposed method.

Weaknesses

1. **Time Complexity Analysis.** Since this method introduces hierarchical optimal transmission, I would like to ask whether there is more time overhead in the inference phase than in other methods. Can you provide relevant experimental comparisons in detail? 2. **More Cases for Visualization.** In order to prove that learned prompt tokens have the ability to capture diverse visual concepts, it seems that Fig. 4 cannot explain the above claim well. I am more curious whether there are multiple prompts that will pay attention to the same visual concepts. Can you provide more cases for analysis? 3. **The different with PLOT[1] and MAPLE[2].** Can you provide a more thorough comparison to illustrate the novelty of your work? 4. **The writing is need to be improved**. It is hard to understand the core idea from the messy introduction. What's more, Sec.2 Background and Sec.4 Related work have a large number of duplicates that exist and can be merged. [1] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models. ICLR 2023. [2] MaPLe: Multi-modal Prompt Learning. CVPR 2023.

Questions

As shown in weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

N/A

The inference time is improved to 129 fps via the parallel computing trick

We acknowledge all the reviewers for their valuable comments and suggestions, which help us improve the submission. We in this paper propose a unified multi-modal prompt tuning framework that aligns M visual prompts and N textual prompts via the prompt-level and token-level OT distance. For the commonly concerned issue of complexity analysis, we have reported the number of learnable parameters and inference time of our ALIGN and baseline methods on Food101 datasets at the first rebuttal stage. The proposed model needs to calculate the hierarchical OT distance for each image and label pair and thus has a higher test time (62 fps). To optimize the test time, we empirically find that these OT operations are independent and can be calculated parallelly. This results in a 129 fps inference time, much faster than the previous result. We thank the reviewers for their constructive suggestions again and believe this inference time improvement widens the practical application of the model. In addition, we would be happy to have further discussions with you, do not hesitate to contact us if you have any questions.

Following up with Reviewer dk2b

Dear Reviewer dk2b, Thanks again for your effort in reviewing our paper and give us a great chance to improve the paper quality. We hope that our response can address your concerns. Considering that the discussion period will end on Aug 21st, we would like to know if you have any other questions about our paper, and we are glad to have a discussion with you in the following days. If our response has addressed your concerns, would you mind considering re-evaluating our work based on the updated information? Best regards, Authors

Response

Thanks for the authors' response, which addressed some of my concerns, e.g., the differences from existing works. Unfortunately, the authors did not report the results on larger sizes of CLIP models, which raises concerns that the proposed method may not work on large models. There's a case that, since larger sizes of foundation models are strong enough, such advanced prompt alignment method fails to yield extra gains compared to existing prompt alignment methods. So I choose to maintain my initial rating.

Response to Rebuttal

Thanks for addressing my feedback; I'm inclined to increase my score from 5 to 6 in favor of the paper's acceptance.