Few-Shot Adversarial Prompt Learning on Vision-Language Models

Summary

This paper proposes a few-shot adversarial training methodology for vision-language models to ensure robustness in downstream tasks of pretrained vision-language models. Specifically, instead of using traditional adversarial training methods, it adapts the TRADES loss, a prominent adversarial training approach, for vision-language models. Additionally, it leverages the structural characteristics of vision-language models by adaptively adjusting the weight of the regularization loss, enabling the model to effectively learn robustness in few-shot learning scenarios. As a result, the proposed methodology allows vision-language models to easily acquire robustness through few-shot learning, demonstrating high performance.

Strengths

The methodology proposed by the authors effectively adapts the TRADES adversarial training method, originally used in other fields (vision), for training vision-language models. Additionally, they apply a uni-modal adversarial-aware mechanism to the TRADES loss. Through further analysis, the authors demonstrate that this mechanism enables the model to distinguish well between natural and adversarial examples, avoiding meaningless learning and promoting meaningful learning, which is highly suitable for few-shot learning. The authors point out that pretrained models often show insufficient performance on natural examples when learning adversarial examples for downstream tasks in few-shot learning scenarios. TRADES appears to be a highly suitable loss function for addressing this issue. Consequently, experiments show that the proposed method achieves superior performance in both clean and robust accuracy compared to existing methods.

Weaknesses

- The motivation does not seem sufficient. Among the drawbacks of previous methods raised by the authors, it is not clearly conveyed how (1) and (2) are addressed and resolved later in the paper. Instead, it might have been better to emphasize in section 3.5 that existing methods or simply applying TRADES can lead to small perturbations that do not promote robust learning, which can be particularly disadvantageous in the constrained context of few-shot learning. For (3), if the issue is that the existing clean accuracy is compromised by training only with adversarial examples during the few-shot learning process, a baseline experiment analyzing the performance when both clean and adversarial examples are used together might have been more insightful. - There is a lack of analysis regarding the proposed Uni-Modal Adversarial-Aware Mechanism. It is unclear why this method is implemented as a multiplicative factor to the KL divergence loss rather than adding a separate loss. The reasons behind the results shown in Figure 2 using this learning method are not clearly explained. While I understand that it is advantageous for learning if clean and adversarial examples are distinctly separated in the embedding space, I am concerned that if clean and adversarial examples are separated in this manner in the final trained model, it might actually reduce robustness. - The analysis of the experiments is insufficient. Although the proposed methodology showed an increase in the overall average performance on the tested datasets, some datasets exhibited similar or even decreased performance compared to existing methods. A deeper consideration and explanation of these observations are necessary.

Questions

Please provide answers to the mentioned points. Additionally, - Were the adversarial examples in Figure 2 generated using PGD or were they generated by maximizing the KL divergence loss? - Are there any comparative experimental results based on full fine-tuning?

Rating

6

Confidence

2

Soundness

2

Presentation

2

Contribution

3

Limitations

Mentioned above

Summary

This paper introduces a novel few-shot adversarial prompt framework for enhancing the adversarial robustness of vision-language models. The authors propose a method that achieves state-of-the-art zero-shot adversarial robustness using only 1% of training data, addressing limitations of existing approaches such as heavy adaptation costs and suboptimal text supervision. The key contribution is a new training objective that improves multi-modal feature consistency while differentiating uni-modal features between natural and adversarial examples. The experiments rely on one multimodal model, but executes experiments on 12 datasets. While the approach shows promise, potential limitations include questions about generalizability across different model architectures. Overall, this paper presents an innovative approach to improving adversarial robustness in vision-language models with promising results.

Strengths

- Novel approach: The paper introduces a few-shot adversarial prompt framework, which appears to be an innovative method for improving adversarial robustness in vision-language models. - Efficiency: The approach achieves state-of-the-art zero-shot adversarial robustness using only 1% of training data, which is a significant improvement in data efficiency. - Multi-modal consistency: The authors propose a novel training objective that enhances the consistency of multi-modal features while encouraging differentiated uni-modal features between natural and adversarial examples. - Practical relevance: The method addresses real-world issues such as heavy adaptation costs and suboptimal text supervision in existing approaches.

Weaknesses

- Generalizability: It's unclear how well this method generalizes across different vision-language model architectures. The authors should provide more information on the range of models tested. - Comparative analysis: While the abstract mentions matching state-of-the-art performance, a more detailed comparison with existing methods would strengthen the paper. Writing: - Line 74: a cross reference would ease reading.

Questions

- Have you thought of how your method performs across different types of adversarial attacks? - What are the limitations of your approach, and are there specific scenarios where it might not perform as well?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed some limitation in the appendix, directing towards more theoretical analysis of the method. In my opinion, long-term robustness could be addressed as well: The paper could have addressed (at least in the future work) how the proposed method performs against evolving adversarial attacks such as adaptive attacks [1]. [1] https://proceedings.mlr.press/v162/croce22a.html

Summary

Adversarial prompt learning on vision-language models has traditionally focused on aligning text with corresponding images to ensure coherence and contextual accuracy. This paper extends this approach by making the image features of natural and adversarial examples distinct while still aligning them with the relevant text descriptions. This novel contribution enhances the model's robustness against adversarial attacks. The authors introduce a new framework that leverages adversarial text supervision to improve cross-modal adversarial alignment. This framework allows for the learning of adversarial prompts, significantly boosting the model’s ability to handle adversarial examples. Remarkably, the proposed method achieves state-of-the-art zero-shot adversarial robustness while utilizing only 1% of the training data, demonstrating both efficiency and effectiveness in enhancing model performance.

Strengths

Novelty: The paper introduces a training objective that enhances the consistency of multi-modal features while encouraging differentiated uni-modal features between natural and adversarial examples. Clarity: The paper is clearly written, with methodologies and results presented in an accessible and comprehensible manner. Significance: The proposed approach is compared with three baseline methods and demonstrates significant improvements in robust accuracy across 11 different datasets.

Weaknesses

Depth of Analysis: The paper introduces a novel training objective that is both interesting and unexpected. However, there is a need for a deeper analysis and explanation of why this proposed method is effective. Understanding the underlying mechanisms and reasons for its success would provide valuable insights and strengthen the paper's contributions.

Questions

When would this few-shot setup particular useful? If we already have LAION and other big dataset?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No.

Summary

This paper addresses adversarial robustness for image classification with VLMs (e.g. CLIP model). To this end, the authors proposed the FAP framework to adapt VLM models by learning prompt tokens in a few-shot manner with adversarial examples. The loss function is proposed to promote accuracy on clean images, to align the text-image embedding interaction across clean and adversarial images and to align the image embedding similarity across clean and adversarial images. Experiments are carried out on image classification tasks, with the few-shot setup on multiple datasets, and the proposed method achieves good results on both base and novel datasets.

Strengths

* The results are indeed better when there are not many training examples. * Extensive discussions are provided in the appendix.

Weaknesses

* The framework architecture is not easy to understand without reading the MaPLe paper. For example, is $P_v$ in line 150 shared shared or separate across different encoder layers? And the modification of $h$'s direction is not clear without reading the appendix. The presentation should be improved and more self-contained. * From Table 1, it seems that most of the gains are coming from the architecture of MaPLe. For the proposed loss in equation 7, the components of them are not carefully ablated. Nitpicking: Line 108: $\theta_T$ -> $\theta_I$

Questions

In Eq 3 and 4, $\bf{t}$ seems to be a list of strings/text prompts (following line 83), how is $cos$ computed on strings? Or are we missing some $z$ here? In Appendix A Algorithm1, are $P_v$ and $P_t$ separate or one is dependent on another? It's not very clear in that codeblock.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Limitations are adequately addressed in the appendix.

Thank you for the reply

My concerns are answered.