UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation

Summary

This paper aims to design a universal unlearnable example generator for image segmentation tasks. Specifically, this paper aims to address three important factors for unlearnable examples in image segmentation:1) data efficiency 2) generation efficiency 3) transferable efficiency. To design such a model, the author makes use of the pretrained SAM model to conduct a min-min optimization problem, which involves two models) a noise generation model based on the pretrained SAM model; 2) a surrogate model training from scratch for minimizing the training loss given the corrupted unlearnable image. To this end, the author conducts comprehensive experiments for a diverse set of downstream tasks to verify the effectiveness of the proposed method.

Strengths

The problem set is of practical meaning, especially in the current situation where private and copyright-protected data are widely used to train large-scale deep learning models. Previous methods to generate unlearnable examples are mainly designed for image classification tasks, which are not directly transferable for image segmentation tasks. To solve this problem, the author proposes to leverage the segmentation foundation model for generating unlearnable examples. The proposed method is easy to optimize and transferable for different downstream tasks.

Weaknesses

The key experiment setup is not clear (see questions). What are the benefits of using the SAM as the backbone for the unlearnable example generation model and the surrogate model, I miss the ablation on this. Besides, what is the benefit of using the SAM pretrained weight for the unlearnable example generation model? Is this crucial for the success of the proposed method?

Questions

It appears that the UnSeg model is trained on the HQSeg44k dataset. Consequently, the unlearned dataset is also HQSeg44k, but with added unlearnable noise. The authors subsequently used the HQSeg44k dataset to fine-tune different models for downstream tasks, including semantic segmentation, instance segmentation, and panoptic segmentation. If my previous explanation is correct, I am concerned about the fairness of this approach. Specifically, there may be a significant domain gap between the fine-tuning dataset and the evaluation dataset. Furthermore, this setup does not demonstrate that the proposed method can generate unlearnable noise for example beyond the training data used for the UnSeg model. But from the caption of Figure 5, it seems that the UnSeg model is employed to generate unlearnable noise for new datasets used in training different downstream tasks, the authors should clarify this in lines 249 to 265. Please clarify the following: On which dataset is the inference of UnSeg conducted (i.e., what training dataset is used for different downstream tasks)? Which dataset is used as the evaluation set for the trained models of different downstream tasks?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitations have been discussed.

Summary

This article uses the powerful SAM to fine-tune a non-learnable examples generator, achieving good protection effects on downstream datasets and models.

Strengths

This article breaks through the concept of unlearnable from classification to segmentation, and the experiment has achieved good results.

Weaknesses

I am mainly concerned about whether the unlearnable samples proposed in this paper on segmentation have much practical application? The author used a relatively strong segmenter to train an unlearnable sample generator, but trained unlearnable samples on a relatively weak segmenter. Is this reasonable in real world?

Questions

Please show experiments on adversarial training on non-learnable examples.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

This paper could not cause any societal impacts.

Summary

Aiming to provide a solution for protecting sensitive/private images, this paper introduces UnSeg, a framework designed to generate unlearnable examples (UEs) to protect images from unauthorized usage in image segmentation models. Utilizing the Segment Anything Model (SAM) and bilevel optimization, UnSeg creates noise that renders images unusable for training segmentation models. The framework is validated through extensive experiments across multiple segmentation tasks, datasets, and architectures (listed in Table 1).

Strengths

- **(Writing)** The reviewer enjoyed reading the paper and the motivation of the idea. Great job, authors! - **(Method)** The reviewer finds the paper to be quite interesting. The idea of the paper is to leverage a foundational model like SAM to add perturbations to a local area that cannot be segmented. Further, the reviewer is impressed with the approach to train a universal perturbation generator that can be readily applied to craft unlearnable noise for any given image in one single forward pass. This is a distinct property of image-agnostic adversarial attacks and its use in this paper’s setting is intuitive. - **(Experiment)** The experiment section is comprehensive. Comparison on three benchmarks on Table 1 shows strong performance of the proposed method. Further, the proposed method also tests the strategy on non-standard benchmarks like Remote Sensing Segmentation and Medical Image Segmentation in Figure 5. The paper also investigates the impact of perturbations on object detection Table 3 and against deployed defenses in Table 4.

Weaknesses

The reviewer did not find major flaws with the paper. - **(Method)** The reviewer feels there will be restrictions due to prevalent memory bandwidth for adoption of this method. The authors can expand on the real-world use cases of the proposed method (in terms of deployment). - **(Method)** The reviewer doesn’t clearly understand the intuition behind the Section “Training Stability and Epsilon Generalization”. The point of segmentation methods being more fine-grained doesn’t justify balancing the perturbations by scaling factor $v$. - **(Experiments)** The reviewer fails to understand why the methods in [48, 61] do not serve as proper baselines. Since the authors compare their method in cross domain/task settings (like in Table 3), these works also serve as good baselines. - **(Experiments)** This paper would definitely benefit from analysis of statistical significance. Localized perturbations may get distributed in different patterns and hence may affect the segmentation performance. - **(Typo)** L223, 220 v -> $v$

Questions

1. Is the method applicable when the model is other than the SAM? 2. Will the method work if the segmentation model leverages contextual knowledge in the input images to create better object masks?

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Authors do not describe any limitations clearly. The reviewer feels the if the surrogate model is not strong/robust enough, the perturbations might not be potent to break the target segmentation models. The reviewer did not find any particular negative societal impact.

Summary

The paper addresses the issue of privacy concerns in training large-scale image segmentation models using unauthorized private data. The authors propose a novel framework called Unlearnable Segmentation (UnSeg) to generate unlearnable noise that, when added to images, makes them unusable for model training. This framework involves training a universal unlearnable noise generator by fine-tuning the Segment Anything Model (SAM) through bilevel optimization on an interactive segmentation dataset. The effectiveness of UnSeg is demonstrated across six mainstream image segmentation tasks, ten widely used datasets, and seven different network architectures, significantly reducing segmentation performance when unlearnable images are used.

Strengths

1. The motivation and the proposed method are impressive. The work addresses three key challenges in generating unlearnable examples: data efficiency, generation efficiency, and transferability. The proposed method effectively employs the Segment Anything Model (SAM) to tackle these challenges. 2. Extensive experiments and ablation studies are conducted to evaluate the proposed method.

Weaknesses

1. The generation of unlearnable examples (UEs) requires object masks, which can be challenging to obtain in common practice. The authors should discuss this issue and consider using bounding boxes or clicks as alternative options. Reporting the results of these alternatives would be beneficial. 2. Why is the surrogate model initialized with the pretrained MAE model weights instead of SAM weights? 3. As shown in Table 5, the model trained with a mixed dataset of clean and unlearnable data achieves similar or even better performance compared to the model trained with only clean data. This suggests that unlearnable data might not reduce performance and could even enhance it. Consequently, network trainers might still collect images without concern, despite the proposed method, unless all images available on the internet are unlearnable samples. 4. Can the authors provide more results from different networks and datasets using the clean-unlearnable mixed training dataset in Table 5? 5. Table 2 only compares one state-of-the-art (SOTA) method. The authors should include more SOTA methods, including attack methods for segmentation and classification, to effectively validate the superiority of the proposed method.

Questions

See Q1 and Q3 in weaknesses, pls.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have introduced the limitations and broader impact of the proposed method.

Author-reviewer discussion

Dear reviewer zWqb and Mhnt, Since the authors provided their responses, please read the responses, respond to them on in the discussion, and discuss points of disagreement if necessary by Aug 13. Best regards, AC

Your further feedback is greatly appreciated.

Dear Reviewer zWqb, Thank you for your initial comments. We understand your concern about the practical application of our work and have prepared a detailed response to address it. Please have a look at our response and kindly let us know if it addresses your concerns. Your further feedback is valuable to us, and we hope to resolve any remaining issues before the rebuttal period ends. Thanks again for your time.

Concerns are addressed.

The authors have addressed my concerns, and I decide to raise my score.