In an era of widespread web scraping, unlearnable dataset methods have the potential to protect data privacy by preventing deep neural networks from generalizing. But in addition to a number of practical limitations that make their use unlikely, we make a number of findings that call into question their ability to safeguard data. First, it is widely believed that neural networks trained on unlearnable datasets only learn shortcuts, simpler rules that are not useful for generalization. In contrast, we find that networks actually can learn useful features that can be reweighed for high test performance, suggesting that image protection is not assured. Unlearnable datasets are also believed to induce learning shortcuts through linear separability of added perturbations. We provide a counterexample, demonstrating that linear separability of perturbations is not a necessary condition. To emphasize why linearly separable perturbations should not be relied upon, we propose an orthogonal projection attack which allows learning from unlearnable datasets published in ICML 2021 and ICLR 2023. Our proposed attack is significantly less complex than recently proposed techniques.
Paper
Similar papers
Peer review
Summary
This paper comprehensively evaluated existing unlearnable examples and showed a surprising result. Unlearnable examples aim to prevent the model from learning useful features from the data. However, results show that several methods that networks actually can learn useful features. This is revealed by applying an existing feature reweighting method. Such results indicate that "privacy" might not be preserved. This paper also demonstrated that the recent findings of linear separation in unlearnable examples are not a necessary condition and an orthogonal projection attack can break class-wise linear separable unlearnable noise.
Strengths
- By using an existing method of Deep Feature Reweighting (DFR), the finding is very interesting and new in this field. This paper has comprehensively evaluated the existing different methods for generating unlearnable examples. It is surprising to see model can "actually" learn useful features on several unlearnable methods. Interestingly, the error-minimizing does not. Although the exact cause remains unknown, such findings provide valuable insights for future works. Additionally, this also indicates that future works should consider DFR as a reasonable evaluation method for unlearnable examples. - Based on recent work, Yu et al. [34] show that unlearnable examples rely on linearly separable noise. This paper extended the analysis to a wide range of different generation methods. Results show that although unlearnable examples are commonly induced linear separable noise, one particular method, Autoregressive perturbations, does not fall into this category. This indicates that linear separable noise is not a necessary condition. These results provide insights for future works that there could be more effective non-linear separable unlearnable noise. - The presentations in this work are very good. It covered comprehensive related works, motivations, challenges and limitations in unlearnable examples.
Weaknesses
My main concern is the Orthogonal Projection attack. Although it is technically sound, it is unclear what is the main contribution or the insight of this proposed method. - Class-wise noise is known to be easily detected in unlearnable examples. Considering an additive noise, Yu et al. [34] show averaging across classes can easily expose the class-wise noise. One trivial solution that might consider is to subtract the noise for each class. Does an Orthogonal Projection attack more effective compared to this trivial solution? For Orthogonal Projection attacks, does it have any effect on the sample-wise setting? - Compared to adversarial training (AT), the main benefit is efficiency. However, AT has other desired properties, such as adversarial robustness and learning robust features. As for effectiveness, it seems to be no significant benefit of the Orthogonal Projection attack. For OPS or OPS+EM, the constraint is a single pixel, while AT evaluated in this paper is $L_\infty$, changing AT constraint might be effective against the OPS-based method.
Questions
See the weaknesses section.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
The authors adequately addressed the limitations and potential negative societal impact.
Summary
This paper suggests that DNNs can learn useful features from unlearnable datasets and provides a counterexample, demonstrating that linear separability of perturbations is not a necessary condition. They propose the Orthogonal Projection method to recover unlearnable datasets.
Strengths
1. A new method called Orthogonal Projection is proposed to recover unlearnable datasets. On class-wise unlearnable datasets, this method achieves similar (or even betters) performance compared to adversarial training. Moreover, it is much faster than adversarial training. 2. This work suggests the risk of using class-wise perturbations to craft unlearnable datasets.
Weaknesses
1. For the claim "DNNs can learn useful features from unlearnable datasets", your experimental evidence is not sufficient and the logic is not very solid. See Question 1. 2. Though the intuition of Orthogonal Projection is clear, the purpose of its mathematical form is still ambiguous. Especially it cannot explain why this method is good at treating class-wise perturbations rather than sample-wise ones. See Question 2. 3. Sections 4.2, 4.3 and 4.4 have weak connections with each other, appearing as if they are independent from one another. The results in sections 4.2 and 4.3 are not interesting enough.
Questions
1. About weakness 1: 1.1 In Figure 2, the performance of DFR on different unlearnable datasets is not consistent. At the last epoch, DFR in (b, d) is even worse than randomly initialized baseline. For certain datsets, such as Random noise and AR, the peak in early epochs is a property of these datsets, not your DFR method. 1.2 I don't think the increase of test accuracy caused by DFR can support you claim because you retrain the linear classifier using clean data. Don't forget that your DFR method improve the accuracy of randomly initialized baseline from $10\%$ to $35.79\%$. However, you can not conclude that a randomly initialized feature extractor learns useful features. 2. About weakness 2: 2.1 In the algorithm of Orthogonal Projection, the recovered data $X_r$ is ensured that $W^T X_r=0$. What does this equation mean? How does it work for recovery? 2.2 What is the reason that Orthogonal Projection is not good at recovering sample-wise perturbations? 2.3 It is mentioned in Line 262 that perturbed images of sample-wise Unlearnable Examples and Adversarial Poisoning are not linearly separable though perturbation themselves are linearly separable. Does this illustration contradict to the claim in Line 216 "learning the most predictive features in an unlearnable dataset can amount to learning the perturbation itself"? 2.4 Please give experimental evidence of linear (in)separability of images perturbed by those class-wise and sample-wise attacks.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
As mentioned in the paper, orthogonally projection removes K dimensions from the data manifold, where K is the number of classes in a dataset. While this may not be a problem for high-resolution images with tens of thousands of dimensions, this detail could impact applicability for low-resolution datasets. Also see the weakness.
Summary
This paper conducts an analysis of the properties of unlearnable dataset methods to evaluate their potential for future viability and security assurances. It is demonstrated that neural networks possess the ability to learn generalizable features from unlearnable datasets, while also suggesting that image privacy may not be effectively preserved. Additionally, a counterexample is provided to challenge the widely held belief that unlearnable datasets induce learning shortcuts through the linear separability of added perturbations. To address this issue, an orthogonal projection attack is proposed, which enables learning from various unlearnable datasets. The results of this approach demonstrate that linearly separable perturbations should not be relied upon.
Strengths
1. The originality of the paper is good, as it gives us a different view to unlearnable examples and poisoning attacks by (a). Demonstrating that neural networks possess the ability to learn generalizable features from unlearnable datasets (b). Giving a counterexample is provided to challenge the widely held belief that unlearnable datasets induce learning shortcuts through the linear separability of added perturbations (c). Proposing a new evaluation framework and a novel attack method to assess the viability and security promises of unlearnable datasets. Overall, The paper's findings challenge some widely held beliefs about unlearnable datasets and provide insights into their limitations and potential vulnerabilities 2. The proposed method can achieve good results on some of unlearnable tasks (class-wise) and is more effective than adversial training. 3. This paper is well-writen
Weaknesses
1. The motivation for the paper does not align to me. I hope that the proposed method is motivated by the findings (Sections 4.2 and 4.3), however, the authors only illustrate the relationship between linearly separable perturbations and the proposed orthogonal projection method, making Section 4.2 a superfluous. There should be more discussion on the relationship between the findings and the methods presented in Section 4.4. 2. The author uses the DFR method to prove that DNN can learn useful features, but it is not convincing to me. The improvement in accuracy may simply come from using cleaner samples. (eg Figure 2, pictures (b), (c)). Also, I think the act of loss should also be involved in this part. There should be more references to why "the higher the test accuracy after DFR, the more likely the model will pick up on private image features present in the original clean data". 3. Although the authors give a counterexample to challenge the commonly held view that non-learnable datasets induce learning shortcuts by adding perturbed linear separability, it lacks theoretical analysis. 4. The method only achieves satisfactory results on Class-Wise perturbation, but the generalization of the method is not enough. In contrast, adversarial training has better generalization ability. 5. Limited evaluation: Evaluation requires more datasets.
Questions
1. The correlation between the method proposed in Section 4.4 and the findings in Sections 4.3 and 4.2, particularly 4.2, is of interest. 2. Can additional sources be provided to corroborate the assertion that "the higher the test accuracy after DFR, the more likely it is that the model has earned private image features present in the original clean data"? 3. The efficacy of the proposed method on other datasets merits investigation. It is recommended that further experimentation be conducted. 4. The dissimilar trends displayed by (a) (d) and (b) (c) in Figure 2 require explanation.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Yes, the authors adequately addressed the limitations. However, in order to further enhance and augment the credibility and validity of their work, it is suggested that they provide a more elaborate and comprehensive explanation regarding the various methodologies and motivations that were employed in their research process. Additionally, it would be highly beneficial if they could include a greater number of experiments and references within their work, as this would serve to substantially bolster and fortify the robustness and soundness of their findings and conclusions.
Summary
This paper studies the problem of the actual learnability of unlearnable datasets. Specifically, the authors have demonstrated that unlearnable datasets that are generated by existing methods can actually be used to learn generalizable features. In addition, the authors show that it is not necessary to make poisons linearly separable for achieving effective poisoning effects. Furthermore, based on the fact that most existing poisoning methods rely on linear separability, the authors propose a simple yet effective countermeasure to recover clean images for training.
Strengths
- It is valuable to revisit existing work in a popular field like data poisoning. - The paper is very well written, with a thorough review of related work and sufficient descriptions of technical details. - Experiments are extensive in terms of the number of models, datasets, settings, compared baselines, and so on. - Several visualizations are provided to help support new findings.
Weaknesses
I very much appreciate the paper, and there are no major weaknesses but minor ones about tuning down some claims: 1. This paper is not the first to find a counterexample to the linear separability hypothesis considering that previous work has intentionally relied on ideas beyond linear separability for generating unlearnable datasets [a]. 2. The orthogonal projection method is not the only simple yet effective countermeasure considering [b], where the simple image compression-based method, ISS, is applied to remove poisoning perturbations. About the results, it seems that ISS is more globally effective than the orthogonal projection method. In addition, It would also be appreciated if diffusion models [3] and error-maximizing augmentation [20] can be compared. These two methods are also conceptually simple because they rely on either a per-trained (diffusion) model for pre-processing or data augmentation (that may not necessarily be adversarial). [a] Is Adversarial Training Really a Silver Bullet for Mitigating Data Poisoning? ICLR 2023 [b] Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression. ICML 2023
Questions
N/A
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
Yes.
Thanks for the clarification
Thanks for the clarification. My concerns have been addressed. I very much appreciate this paper.
Thanks for the rebuttal!
My concerns are well addressed. The authors are encouraged to incorporate the above discussions into the final version.
While your previous explanation helped address some of my questions, there are still a couple major issues that need to be better clarified: 1) It would be helpful to see a stronger link between the orthogonal projection method proposed in Section 4.4 and the results from Sections 4.2 and 4.3, especially 4.2. The proposal seems like it should be motivated by those earlier findings, but right now the paper only shows the connection to linearly separable perturbations, making Section 4.2 seem unnecessary. Please illustrate more clearly how the observations in 4.2 and 4.3 led to the proposed technique. 2) For the weakness 2, the evaluation should include more experiments and metrics beyond just test accuracy, such as loss over training, Relying solely on test accuracy doesn't fully support the claim. Including additional graphs would provide a more thorough analysis.
Clarifications and new metrics in response to comment
> Please illustrate more clearly how the observations in 4.2 and 4.3 led to the proposed technique We can see how the paper’s organization could suggest a deeper connection between our findings in 4.2 and 4.3 and the Orthogonal Projection method. But, as we state in the abstract and introduction, the goal of our paper is “make a number of findings that call into question [Unlearnable Datasets’] ability to safeguard data.” Each section of the paper (4.2 to 4.4) is a finding that, we believe, can be viewed on its own and can inform the poisoning community about fundamental issues of current unlearnable dataset methods. For example, Unlearnable Datasets can be seen as not suitable to protect data because one can learn generalizable features from poisoned data (Section 4.2). On the other hand, Unlearnable Datasets can be seen as not suitable to protect data because class-wise perturbations are still commonly used and we can develop a method to find and remove them (Section 4.4). In both cases, model parameters are optimized using unlearnable data – in Sec 4.2 the optimization yields generalizable features, while in Sec 4.4 the optimization yields the class-wise perturbation. We have added additional clarifications to the introduction, we would be open to suggestions as to re-organization of the findings to minimize confusion. > the evaluation should include more experiments and metrics beyond just test accuracy, such as loss over training Following your advice, we have measured the train and test loss of each poison checkpoint after DFR and compared it to the loss of a randomly initialized model after DFR. As a reminder, both these losses are computed when evaluating poisoned checkpoints on clean train/test data. The following table performs the same experiment as in Table 1, Section 4.2, where we investigate what poisoned models learn. We use the same random subset of 5,000 clean CIFAR-10 train samples from the paper for DFR. | Train Data | Min DFR Train Loss | Min DFR Test Loss | |---------------------------------------|-----------------------|----------------------| | None | 2.625 | 2.379 | | Unlearnable Examples [10] | 1.777 | 1.798 | | Adversarial Poisoning [7] | 0.966 | 1.036 | | AR [26] | 1.408 | 1.531 | | NTGA | 1.342 | 1.391 | | Robust Unlearnable [27] | 1.789 | 1.790 | | LSP [34] | 1.705 | 1.675 | | OPS+EM [33] | 1.857 | 1.869 | | (CW) OPS [33] | 1.669 | 1.697 | | (CW) Unlearnable Examples [10] | 2.043 | 2.037 | | (CW) Regions-4 [25] | 1.669 | 1.706 | | (CW) Random Noise | 1.170 | 1.352 | In the first row, we report the train and test loss of a randomly initialized RN-18 after DFR. In the subsequent rows, we report min train/test loss of poisoned checkpoints after DFR. We find that the trends when analyzing the loss follow the trends when analyzing test accuracy: Firstly, Adversarial Poisoning, AR, NTGA, and (CW) Random Noise achieve lowest losses – and accordingly those poisoned checkpoints also have the highest DFR test accuracy in Table 1. Secondly, and more interestingly, we find that **all** poisoned models have a lower loss than the finetuned (DFR) random initialized model. This reinforces our claim that models learned useful features from poisoned data. Interestingly, (CW) Unlearnable Examples come closest to the expected random chance loss of -log(1/10)=2.302. This suggests (CW) Unlearnable Examples are best at corrupting network weights from finetuning. Note that train loss is slightly lower than test loss in many cases because train loss is computed over the entire train set (which includes the subset of 5k samples we used for DFR). Thanks again for your suggestions which have made our work more comprehensive. We have produced two additional plots (DFR train loss vs epoch and DFR test loss vs epoch), and a new table which we will include in our camera-ready version. We would appreciate it if you would consider revising your score in light of our response. And do let us know if you have any additional questions.
I appreciate the efforts made by the authors to answer my questions. Firstly, your clarification helped me understand why your proposed orthogonal projection method is effective, especially in defending against class-wise poisoning. Taking CIFAR-10 as an example, we can approximate that the 10 column vectors of Q represent 10 class poisons. Your method precisely makes the processed images orthogonal to that 10-dimensional subspace. I find this idea reasonable. However, this also highlights the limitation of the method, which is the inability to guarantee the effectiveness for sample-wise poisoning, especially considering some recently proposed defense methods [a, b] that achieve better defense results. Perhaps focusing only on defense against class-wise poisoning would be helpful. Secondly, I still believe that Sections 4.2 and 4.3 are not sufficiently interesting and have weak connections to the most important Section 4.4 in the overall article. Therefore, I decided to keep the rating. [a] Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression. ICML 2023 [b] Learning the Unlearnable: Adversarial Augmentations Suppress Unlearnable Example Attacks. arXiv 2023.
Clarifications on importance of findings in response to comment
> limitation of the method, which is the inability to guarantee the effectiveness for sample-wise poisoning, especially considering some recently proposed defense methods [a, b] Given that class-wise poisons are still being developed in numerous works [10, 33, 34, 18], we believe research would benefit from our novel benchmark attack (Orthogonal Projection). We believe that without knowledge of our results, class-wise poisons could continue to be developed without a strong baseline to compare against. In fact, one of the reasons we developed the Orthogonal Projection method was because we could not find a good explanation for why papers opted for sample-wise poisons when class-wise poisons were more effective [10]. Our work demonstrates that removing a class-wise perturbation is not as simple as taking the average class image (see rebuttal pdf), and that perturbations can be recovered, as we illustrate in Figure 3. While ISS [b] is effective, there is little theoretical evidence to suggest that this effectiveness is “guaranteed.” While the authors explain that “grayscale compression is used to eliminate low-frequency shortcuts, and JPEG compression is used to eliminate high-frequency shortcuts,” a DCT transform is never performed to analyze whether this is occurring or not. On the other hand, vector projections and orthogonal subspaces can be reasoned about for images theoretically, and tested empirically as we do in Section 4.4 and Appendix A.3. Compared to ISS [b], our Orthogonal Projection method produces higher test accuracy on the recently published class-wise OPS poison (87.94% (ours) vs 85.16% [b]). [10] Unlearnable Examples: Making Personal Data Unexploitable, ICLR 2021 [18] Learnability lock: Authorized learnability control through adversarial invertible transformations, ICLR 2022 [...] remaining citations are from paper references > Sections 4.2 and 4.3 are not sufficiently interesting Section 4.2 presents a new way of analyzing learned representations of poisoned models. Deep feature reweighting allows us to probe poisoned models for useful features and evaluate unlearnable dataset methods in a new way. We find it interesting that some unlearnable datasets produce checkpoints that progressively improve test accuracy during poison training. Section 4.3 presents a counterexample to the linear separability hypothesis [34]. Without knowledge of this result, one could be inclined to believe that data poisoning is extremely simple, relying on only linearly separable perturbations despite a wide variety of different optimization objectives and theory used to generate/optimize the poisons. We find it interesting that newer methods [26] can be more complex (not linearly separable) and still work as poisons. This result could inspire future research on non linearly separable perturbations, given that necessary conditions for unlearnable datasets remains an open problem. > weak connections to the most important Section 4.4 We can see how the paper’s organization could suggest a deeper connection between our findings in 4.2 and 4.3 and the Orthogonal Projection method. But, as we state in the abstract and introduction, the goal of our paper is “make a number of findings that call into question [Unlearnable Datasets’] ability to safeguard data.” Each section of the paper (4.2 to 4.4) is a finding that, we believe, can be viewed on its own and can inform the poisoning community about a fundamental issue of current unlearnable dataset methods. For example, Unlearnable Datasets can be seen as not suitable to protect data because one can learn generalizable features from poisoned data (Section 4.2). On the other hand, Unlearnable Datasets can be seen as not suitable to protect data because class-wise perturbations are still commonly used and we can develop a method to find and remove them (Section 4.4). — Thank you again for taking the time to read and discuss our work with us. We appreciate your insights and have made additional paper edits following this response, in addition to the new table from your previous suggestion in this thread. Are there other specific suggestions that, if addressed, would potentially warrant a reconsideration of the current rating? Your guidance would greatly assist us in refining the paper's content.
Decision
Accept (poster)