Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models

Summary

The paper focuses on enhancing the robustness of CLIP models against adversarial perturbations. The approach utilizes saliency maps generated by the inner products between image and text embeddings. Two regularization terms are introduced: the first aims to align the saliency maps of the original and its adversarial examples, while the second seeks to match the saliency maps of the original example between the pre-trained CLIP model and the target model. Experimental results indicate that fine-tuning CLIP with these two regularization terms improves its robustness to attacks.

Strengths

1. The paper is well-written and organized, with clear descriptions of the motivation and methods. 2. To the best of my knowledge, enhancing CLIP robustness by aligning the saliency maps produced through inner products between image and text embeddings, termed "Text-Guided Attention" in the paper, is a novel approach. 3. It is good to see the detailed ablation analysis.

Weaknesses

1. The abstract claims that the goal is to enhance both the generalization and robustness of the CLIP model; however, the experimental results do not fully support this assertion. Specifically, regarding generalization, Table 2 shows a significant decrease in clean accuracy compared to the original pre-trained CLIP model. I kindly ask the authors to use the terms "generalizability" and "adversarial robustness" more carefully within the paper. Typically, the generalizability of CLIP models refers to their zero-shot classification performance across different datasets. It may be argued that the proposed method improves the adversarial robustness of CLIP models while maintaining relatively good generalizability compared to previous adversarial training methods. However, the authors must acknowledge the decrease in generalizability compared to original CLIP model as clearly evidenced by the results in Table 2. 2. Since the training uses adversarial examples produced by PGD, it is not surprising that the proposed method results in a model with good robustness to PGD attacks, as shown in Table 1. However, when the evaluation comes to AutoAttack, the advantage of the TGA-ZSR over the previously best method is marginal as shown in Table 3. Could the authors provide explanations for this point, and could they demonstrate the robustness of the resulting models to more attacks?

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

4

Contribution

2

Limitations

The primary limitation of the paper is that the experimental results presented do not sufficiently support its claims. I recommend that the authors either revise the claims to align more closely with the data presented or extend the experimental section to provide additional evidence supporting their assertions.

Summary

This work studies the robustness to adversarial samples for CLIP. Inspired by an observation that adversarial perturbations induce shifts in text-guided attention, the work proposes a simple yet effective approach to improve the zero-shot robostness, i.e., align the text-guided attention of clean samples and adversarial samples. The state-of-the-art performance across 16 datasets demonstrate the effectiveness of the proposed method.

Strengths

This work focuses on an interesting topic, which involves the robustness of VLMs. To improve the zero-shot robustness of CLIP, the work proposes a text-guided attention-based method, which is simple but intuitive and effective. Extensive experiments on 16 datasets demonstrate the effectiveness of the proposed method. Additionally, the paper is well written and the experiments are solid.

Weaknesses

I would like to see more comparison with other types of attention, e.g., vision-only self-attention. Does the proposed method works because of the attention mechanism or the text guidance?

Questions

1. How do you utilize the class token in the ViT backbone of CLIP? 2. In Tables 13 and 14, why the losses L1 and L2 have very different performance?

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The paper has discussed the limitations and potential impacts of the work.

Summary

This paper proposes an approach to improve the adversarial robustness of vision-language models while maintaining performance on clean images. The key idea is to aligns the attention maps of adversarial examples with clean examples. Extensive experiments demonstrate the effectiveness of the proposed method in zero-shot adversarial robustness across multiple datasets while maintaining high clean accuracy.

Strengths

1. The proposed method is simple, straightforward and effective. It leverages text-guided attention to significantly enhance zero-shot adversarial robustness while maintaining clean accuracy across diverse datasets. 2. The experiments are comprehensive. The results on 16 datasets demonstrate consistent improvements over baseline methods.

Weaknesses

1. The method is only demonstrated on CLIP, raising questions about its applicability to other vision-language models or architectures. 2. The method introduces additional hyperparameters without a clear strategy for tuning them across different datasets or tasks. 3. No error bars or statistical significance tests reported for the experimental results.

Questions

1. How sensitive is the method to the choice of hyperparameters? Is there a principled way to select these across different datasets or tasks? 2. Are there any scenarios or types of adversarial attacks where this method performs poorly? What are the limitations of this approach?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors did not explicitly address the limitations such as the types of adversarial attacks where this method performs poorly.

Summary

The paper proposes a framework, Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR), to enhance the robustness of vision-language models (VLMs) against adversarial attacks. The proposed method incorporates two modules: the Attention Refinement module and the Attention-based Model Constraint module. The Attention Refinement module aligns the text-guided attention of adversarial examples with clean examples, while the Attention-based Model Constraint module maintains the model’s performance on clean samples.

Strengths

1. The use of text-guided attention allows to enhance zero-shot robustness. 2. Extensive evaluation of the approach was conducted across diverse datasets. 3. The proposed method outperforms current state-of-the-art techniques in both zero-shot robust accuracy. 4. The paper provides a thorough analysis of the impact of adversarial attacks on text-guided attention, offering insights into the model's decision-making process. 5. The method improves robustness without significantly sacrificing performance on clean data, achieving a favorable trade-off.

Weaknesses

1. The incorporation of text-guided attention mechanisms and multiple modules may increase the complexity of implementation and computational overhead. 2. Although the authors claim zero-shot robustness in Vision-Language Models, the experiments and methods only target the CLIP model. 3. Missing limitation in Section 5.

Questions

1. What is the additional computational overhead introduced by the Attention Refinement and Attention-based Model Constraint modules? How does this impact training and inference times compared to baseline methods? 2. How well does the proposed method generalize to other types of vision-language tasks beyond image classification, such as image captioning or visual question answering? 3. Why is the proposed method fine-tuned on Tiny-ImageNet rather than ImageNet? Both TeCoA [1] and FARE [2] are trained on ImageNet. [1] Mao, C., Geng, S., Yang, J., Wang, X., & Vondrick, C. (2022). Understanding zero-shot adversarial robustness for large-scale models. [2] Schlarmann, C., Singh, N. D., Croce, F., & Hein, M. (2024). Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A