Transferable Adversarial Attacks on SAM and Its Downstream Models

Summary

This work discusses an interesting security issue of deploying a model fine-tuned on a large foundational model in private downstream applications. It proposes a universal meta-initialized and gradient robust adversarial attack (UMI-GRAT) to break the powerful SAM and its various downstream models, without requiring prior knowledge of the specific downstream task and data distribution. The author explores the challenges associated with threating transfer-based adversarial attack without the task-related prior knowledge and provides the theoretical insights on the deviation in updating the adversarial perturbation when using the open-sourced model as the surrogate model. An extensive evaluation of UMI-GRAT's performance, transferability, and efficiency was conducted across five datasets and three different downstream tasks (medical image segmentation, shadow segmentation and camouflaged object segmentation), demonstrating the high effectiveness of the UMI-GRAT approach.

Strengths

1. This work discusses a critical adversarial issue of deploying large foundation model in real-world applications and for the first time considers a more challenging and practical scenario where the adversarial attacker breaks SAM and its downstream models in the absence of prior knowledge on the task and data distribution. 2. This work provides the in-depth analysis on the challenge of threating the transferable adversarial attack via open-sourced SAM and proposes the corresponding theoretical insights and solution. 3. The work establishes a detailed experimental framework and the proposed UMI-GRAT shows superior performance on misleading various SAMs’ downstream models compared with previous methods, which serve as a preliminary exploration for future research.

Weaknesses

1. It’s recommended to give more comprehensive analysis of the UMI noise, including the size of the natural image dataset and the effect of various hyperparameters. 2. There are more metrics such as $E_\phi$, $F_\beta^\omega$ in the camouflaged object detection task. It would be beneficial if the author could provide further data pertaining to these evaluation metrics to enrich the analysis.

Questions

1. In Figure 3, the peak cosine similarity of the generated adversarial perturbation is observed between the 20th and 30th iteration. So will increasing the iterative step of generating adversarial perturbation enhance the transferability? 2. Model ensemble is an effective method to enhance the adversarial attacks’ transferability. Will the ensemble of different SAMs benefit the UMI-GRAT?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The experimental results are all based on SAMs and their downstream models. It will provide more valuable insights if expanding the scope of analysis to assess whether this adversarial threat also applies to other large foundation models.

Summary

In this paper, the authors present a new approach for adversarial attacks on Segment Anything Model (SAM)-based downstream models, addressing the challenge of attacking without prior knowledge of the downstream task or data distribution. Their key contribution is a universal meta initialization-based algorithm that exposes inherent vulnerabilities in the foundation model. The authors also introduce a gradient robust loss, which simulates uncertainty through gradient-based noise augmentation. This loss is derived from a theoretical formulation of adversarial update deviation between the open-sourced SAM and its fine-tuned downstream models. The authors provide an analytical demonstration of how their proposed method enhances attack transferability. The effectiveness of their approach is thoroughly validated through comprehensive experiments.

Strengths

Originality: This is the first work to explore the feasibility of adversarially attacking various downstream models fine-tuned from the Segment Anything Model (SAM). The introduction of a universal meta initialization-based algorithm to uncover intrinsic vulnerabilities in foundation models is both effective and efficient. Additionally, the formulation of adversarial update deviation and the proposal of a gradient robust loss that simulates uncertainty with gradient-based noise augmentation further enhance the transferability of adversarial examples. Quality and Clarity: The writing is generally clear but has room for improvement. The methodology and results are well-structured, though some technical sections could benefit from additional clarification. Significance: This work is highly significant given the increasing prevalence of foundation models like SAM. The proposed methods for enhancing attack transferability have important implications for AI system security and could influence future directions in both offensive and defensive strategies in adversarial machine learning for SAM.

Weaknesses

1 - My major concern is related to the novelty of the proposed approach. Although I agree that this is the first work in the context of SAMs, the main components, such as downstream agnostic adversarial examples and meta learning-based fast initialization, have already been proposed in the literature. 2 - The authors, in line 45, briefly highlight downstream agnostic examples in just one line. They should clarify in the related work section how their work is different from references 55 and 56 of the main paper, beyond just applying it to SAM. Similarly, another related work that the authors missed is [1] (given below), in which the generated adversarial examples are agnostic to downstream tasks. 3 - Similarly, the authors did not mention any work related to meta-learning-based adversarial examples in the paper. There are multiple works that use meta-learning to craft universal adversarial examples, such as [1, 2] below. The authors use these meta-learning-based methods for initialization of adversarial examples, but this has already been explored in [3] below. The authors should mention these meta-learning-based approaches in their paper and discuss how their method is different from these approaches, beyond just the application to SAMs. 4 - It is not clear to me when the authors claim in line 8 that they are attacking "without accessing the downstream task." What is the task here? Is it not the segmentation task? In [1], their task-agnostic adversarial examples are effective against classification, detection, and segmentation. Since the downstream task here is segmentation-based, is it not obvious what the task is? Please clarify this. 5 - The authors should include some specific aspects of SAM to make their attack more unique. Currently, they are utilizing the SAM image encoder, which, in my opinion, is not much different from the previous works listed below. 6 - For experiments, why have the authors compared their method with intermediate-level feature-based approaches? They should also compare it with different downstream agnostic adversarial approaches as listed below. 7 - In Equation 8, how did the authors choose the threshold lambda? [1] A Self-supervised Approach for Adversarial Robustness (CVPR-2020) [2] Learning to Generate Image Source-Agnostic Universal Adversarial Perturbations (IJCAI22) [3] Meta Adversarial Perturbations (AAI2022-Workshop) [4] Adversarial Initialization with Universal Adversarial Perturbation: A New Approach to Fast Adversarial Training

Questions

Please see the weakness section. While the paper presents an approach to attacking SAM-based downstream models, it largely combines existing methods rather than introducing new techniques. The current strategy, though effective, does not fully exploit SAM's unique architecture.

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes.

Summary

This paper proposes an adversarial attack against fine-tuned derivatives to a publicly available foundation model, such as the Segment Anything Model (SAM). In the proposed threat model, attackers can potentially manipulate these downstream models even without knowing the specific task or data they are used for. Under this threat model, proposes a new attack method called UMI-GRAT (Universal Meta-initialized and Gradient Robust Adversarial Attack). Through a bi-level optimization procedure, this method leverages the information from the open-source SAM to create adversarial examples that can fool mislead the original SAM and its fine-tuned versions. Finally, this paper demonstrates the effectiveness of the proposed UMI-GRAT attack against SAM through extensive experiments.

Strengths

1. The paper is motivated by real-world safety concerns for fine-tuning a public foundation model on private domain-specific datasets. 2. The figures and tables are well-polished and generally reflect the overall message of the paper. 3. The proposed UMI-GRAT attack method is unique and backed by theoretical analysis.

Weaknesses

1. The effectiveness of the proposed attack is only demonstrated by attacking the SAM model. However, more experiment settings (e.g. against pretrained MAE models) are warranted to demonstrate the generalizability of the proposed attack.

Questions

1. How does the domain gap between the natural image dataset used to obtain the universal adversarial trigger and the downstream dataset influence the effectiveness of the attack? 2. How effective is the proposed method against adaptive defense? For example, if the downstream victim model has gone through adversarial training, how effective would the adversarial trigger obtained on the unguarded pretrained SAM be against the guarded victim model?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors acknowledge the limitations of this work in the appendix. They candidly acknowledge the limitations in evaluations as the proposed attack is only evaluated against SAM. I appreciate the authors openly acknowledging this limitation.

Summary

This paper investigates the vulnerability of Segment Anything Model (SAM) and its downstream models to transferable adversarial attacks. The authors propose a novel attack method called Universal Meta-Initialized and Gradient Robust Adversarial attack (UMI-GRAT) that leverages the open-sourced SAM to generate adversarial examples effective against fine-tuned downstream models, even without access to the downstream task or dataset.

Strengths

1. The paper tackles a practical and challenging problem of attacking downstream models fine-tuned from a publicly available foundation model without knowledge of the downstream task or data. 2. The proposed UMI-GRAT method is well-motivated and technically sound. The authors provide theoretical insights into the gradient deviation problem and propose a robust solution using gradient noise augmentation. 3. The paper presents extensive experiments demonstrating the effectiveness of UMI-GRAT in attacking SAM and its downstream models

Weaknesses

See Questions.

Questions

1. How does the performance of UMI-GRAT vary with different choices of hyperparameters, such as the perturbation bound ε and the number of iterations in UMI and LGR. Especially, line 278 mentions that the perturbation bound is 10, which is a bit too large. 2. According to the latest benchmark **[R1]**, baselines used in the paper are not SOTA methods. How does it compare with NCS **[R2]**, ANDA **[R3]**, DeCowA **[R4]** and L2T **[R5]**? 3. Is the proposed method a universal transfer attack method? Although this question is mentioned on line 548, can the performance of UMI-GRAT and SOTA be compared under a general transfer attack test setting? 4. UMI-GRAT is a gradient-based attack method. How does the proposed method perform when the model has a certain robustness (such as adversarial training)? --- **[R1]** Devling into Adversarial Transferability on Image Classification: A Review, Benchmark and Evaluation. **[R2]** Enhancing Adversarial Transferability Through Neighborhood Conditional Sampling. **[R3]** Strong Transferable Adversarial Attacks via Ensembled Asymptotically Normal Distribution Learning. CVPR. 2024. **[R4]** Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warping. **[R5]** Learning to Transform Dynamically for Better Adversarial Transferability

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are discussed in the Appendix.

Summary

In this paper, the authors propose an adversarial attack method that can contaminate downstream tasks from the perspective of adversarial transferability. They address the problem that SAM models do not have similar optimisation routes after fine-tuning for different downstream tasks by designing universal meta initialization. In this paper, the authors address the problem that SAM models do not have similar optimisation routes after fine-tuning for different downstream tasks by designing UMI noise. The authors introduce the idea of meta-learning to allow their algorithm to quickly adapt to different situations, i.e., downstream tasks.

Strengths

1. The theoretical part of this paper is detailed, the experiments are sufficient. The comparison with other methods shows the sophistication of their approach. 2. The attacks proposed in this paper are novel. It contributes to the topic of attacking downstream tasks of large models. A discussion on adversarial transferability is introduced under this topic.

Weaknesses

1. The readability of the Methodology section of this article is somewhat poor. The authors define the problem to be solved through the form of propositions. Similarly, if the authors could summarise the formulas as Theorem and put the proof process (both formulas and reasoning) specifically in the supplementary material, it would make the article more coherent. 2. The randomness of the experimental results is unknown. I understand that due to the larger computational effort, it is not practical to report error lines on all major experiments. But it would be better for the authors to report a set of randomness on a smaller dataset and simpler settings, which will influence the reviewers' opinion of the results of this method.

Questions

Two questions listed, see Weaknesses for details. Note that if the authors can demonstrate the randomness of their algorithms, that will help to get a higher rating.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors correctly list the implications of their work for the use of large models such as SAM in downstream tasks.

Thanks for Your Response & Missing PDF

Thank you for your detailed responses and follow-up experiments! However, I couldn't find the PDF file you submitted. If you could provide me with a pointer to the revised PDF, I will make sure to go over it in the upcoming days. Thank you!

Response to the Authors

After carefully reviewing the comments from the other reviewers and the author's rebuttal, all of my concerns have been adequately addressed. Therefore, I decide to raise my score.

Response to the Authors

It looks like the randomness of most of the data is controlled. The authors also promised to fix the mentioned errors in the next draft. Therefore I decided to raise my score.