Efficient Availability Attacks against Supervised and Contrastive Learning Simultaneously

Summary

This paper studies the transferability of unlearnable examples (UEs) across different learning paradigms, i.e., supervised learning and self-supervised contrastive learning. Different from existing works including both supervised UE generation methods and hybrid methods, this paper argues that strong data augmentations with supervised learning are sufficient for unsupervised unlearnability. It then proposes two strong data augmentations equipped UE generation methods AUE (with standard training) and AAP (with adversarial training). The two proposed methods demonstrate certain superiority over the existing methods on low- and high-resolution datasets.

Strengths

1. The proposed methods are extremely efficient than contrastive learning involving UE generation methods. It leverages supervised learning with strong data augmentation to achieve unsupervised unlearnability against contrastive learning. 2. The proposed method is simple but effective. The revealed finding that strong data augmentation is important for contrastive learning is interesting. 3. Extensive experiments confirmed the capability and efficiency of the proposed attacks.

Weaknesses

1. The alignment and uniformity metrics are all from existing works, which lowers the novelty of the finding in Table 1: "contrastive unlearnability is highly related to huge alignment and uniformity gaps". And the two proposed methods have little to do with the two gaps? 2. While the finding that "supervised learning with strong data augmentation can achieve unsupervised unlearnability" is interesting, the two proposed methods have limited technical novelty. In other words, they do not explore the power of data augmentation more systematically. The current mechanism is very ad-hoc. 3. The performance difference (AUE vs AAP) between low- vs high-resolution datasets is intriguing. AAP targets adversarial training exploitation, so why it should perform better or worse than AUE against contrastive learning? Does contrastive learning have anything to do with adversarial training?

Questions

1. Should the worst-case unlearnability contain other learning paradigms rather than just supervised learning and contrastive learning? For example, MAE or autoregressive pretraining. We cannot enumerate all training paradigms. 2. Why data augmentation is so important for obtaining unsupervised unlearnability, what if there are non-data-augmention-based unsupervised learning methods? -------- Post rebuttal: I have increased my rating according to the authors' responses.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The technical novelty of this work is somewhat limited.

Summary

This paper explores efficient availability attacks that target both supervised and contrastive learning algorithms simultaneously. The authors propose two attack methods, named AUE (Augmented Unlearnable Examples) and AAP (Augmented Adversarial Poisoning), which utilize enhanced data augmentations to craft effective poisoning attacks. These methods achieve state-of-the-art unlearnability with reduced computational demands, showing promise for practical applications. The paper evaluates the effectiveness of these attacks across multiple datasets and demonstrates their superiority over existing methods in achieving worst-case unlearnability while handling high-resolution datasets efficiently.

Strengths

- The motivation behind the study is sound, as it addresses an important gap in existing methods, which are unable to achieve unlearnability for both supervised and contrastive learning. This realization underscores the need to explore techniques that can simultaneously achieve supervised and contrastive unlearnability. - The two proposed attack methods, AAP (Augmented Adversarial Poisoning) and AUE (Augmented Unlearnable Examples), are presented as simple yet effective. - The experiments conducted are comprehensive and robust.

Weaknesses

- While the motivation is justified, the technical contributions could be considered direct. Integrating data augmentation into the generation of poisons is a straightforward approach, which may not reflect a non-trivial technical contribution. - The methods lack theoretical analysis, which raises questions about the guaranteed effectiveness of data augmentation in all scenarios. Additionally, there is no explanation of why data augmentation works effectively or if it completely addresses the issue at hand through augmentation alone.

Questions

What is the biggest challenge in this work?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

None

Summary

This paper claims to introduce a threat scenario where the adversary may use contrastive learning to bypass the unlearnable examples that are crafted only for supervised learning. In this threat model, the authors showcase that previous works on unlearnable examples crafted only for supervised learning may fall short of contrastive learning. Based on this observation, the authors propose a new approach that considers both supervised and contrastive learning when generating unlearnable examples. The proposed approach is also efficient due to the augmentation adopted augmentation strategies. Extensive experiments show that the proposed method is effective in this new threat scenario.

Strengths

The paper is clearly presented. The proposed approach is efficient thanks to the augmentation strategies that imitate the CL. As a consequence, AUE and AAP are also easily scalable to practical datasets, like ImageNet subsets. The authors provided a very complete analysis to evaluate the proposed methods, including discussions on limitations and performance under different defenses. It provides a relatively complete understanding of the proposed approaches in different threat scenarios.

Weaknesses

The introduced topic is interesting but also limited. The authors point out that there is a lack of unlearnable approaches that are effective against both supervised learning (SL) and contrastive learning (CL). This observation is not fully novel, since CP, TP, and TUE have similar observations and are effective in most cases against both types of learning. For example, CP is an approach that is designed for contrastive learning but is also effective against supervised learning. I would suggest that the author could tone down the claims that hint at this observation as a novel finding. The main contribution of this work is to employ both SL and CL losses in the generation of unlearnable examples, including augmentations that imitate the CL in the SL scenario. The proposed approaches show better average and worst-case performance, but the improvement is incremental, according to Table 3 and Table 4. I would suggest that the authors could provide a deeper and more detailed analysis of the difference between SL and CL unlearnable examples by conducting more ablation studies on the augmentation strategies. The working mechanism behind the augmentation strategies is of interest to the research community.

Questions

Please motivate the threat scenario where the proposed unlearnable examples are necessary. Further analysis of the augmentation strategies could be clarified.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have discussed most limitations adequately.

Summary

This paper aims to address the issue of effectively conducting availability attacks on both supervised learning (SL) and contrastive learning (CL) algorithms. Specifically, the paper highlights that existing methods fail to simultaneously achieve "unlearnability" for both SL and CL, posing risks to data protection. To tackle this challenge, the paper proposes a novel approach that employs contrastive-like data augmentation within the supervised learning framework to execute effective attacks on both SL and CL.

Strengths

1. The proposed methods achieve better efficiency and Pareto improvement for both the SL and CL tasks. 2. the first work that demonstrates adding contrastive augmentation resilience with an SL-based surrogate can fool CL-based models.

Weaknesses

1. The proposed AUE and AAP are all brittle under diffusion-based purification, which limits the practical usage of unlearnable examples in the real world. 2. Lack of discussion and comparison with recent work [1] that leverages CLIP latent as guidance for crafting transferable and label-agnostic perturbation, which could potentially achieve efficient perturbation crafting for both SL and CL tasks. 3. The results in Table 3 show that the proposed methods did not consistently outperform other methods under different SL and CL settings, which makes the stability of improvement questionable. 4. This work essentially works by crafting perturbation that resilience to augmentations for both supervised learning and contrastive learning, which is a conceptually simple extension of the transformation augmentation technique in REM [2]. One of the fundamental limitations of such a direction is that unauthorized trainer might leverage stronger transformations like super-resolution or diffusion-based augmentation [3] in their training pipeline. Typos: Line 20: Change "particular, Huang et al. [20] reduces" to "particular, Huang et al. [20] reduced." Ref. [1]. One for All: A Universal Generator for Concept Unlearnability via Multi-Modal Alignment, ICML'24 [2]. Robust Unlearnable Examples: Protecting Data Against Adversarial Learning, ICLR'22' [3]. DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentation, ICML'24

Questions

Nowadays, one of the most popular ways to build classifiers is to leverage CLIP as a feature encoder with or without additional head to do classification. Is the perturbation crafted from AUE and AAP transferable to this setting?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Thank you for your detailed response

Thank you for your detailed response. Your response addresses most of my concerns. I will raise my score. But I am still looking forward to seeing the perturbation against DiffAug experiments, since, as you mention, it's a fundamental challenge in this field.

Thanks for the clarification

Thanks. Most of my concerns have been addressed, thus I increase my rating to 6.