We introduce camouflaged data poisoning attacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider clean-label targeted attacks (in which the goal is to cause the model to misclassify a specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of a poisoned dataset.
Paper
Similar papers
Peer review
Summary
The paper proposes a novel attack in machine unlearning. There are two step in the attack process. In the first step, the poisoned data and the camouflaged data are fed into the initial model training, together with the benign data. The machine unlearning process would take out the camouflaged data and the retraining from scratch process will be on the poisoned data and the benign data. Experiments are done to demonstrate the proposed attack.
Strengths
• The authors propose a novel attack in machine unlearning. • Evaluations are done on the label flipping and gradient matching setting for camouflaged data generation, two types of models (SVM and neural networks will multiple different network structures), and three datasets. • The paper is well-written and easy to follow.
Weaknesses
• The paper makes impractical assumptions: o In step 4, the unlearned model is by training the clean ones and poisoned ones. This reviewer fails to understand if the initial training can identify the poisoning effect and require the camouflaged data to mask the poisoning effect, the why the machine unlearning procedure does not offer such capability, especially the authors adopt the retraining from scratch strategy to perform machine unlearning? • The paper makes a few false claims: o This reviewer does not understand why the authors call the proposed approach clean-label when they have a poisoned set generated very similar to adversarial examples: looks the same for human but will be misclassified by the model. Even if the attack does not change the label directly, gradient matching would still change the optimization of the objective. Otherwise, if the label or the intermediate computation results is not changed and only the input data is slightly modified, the so-called attack becomes adversarial training. o The authors are making speculations which are not true. The authors keep emphasizing the proposed approach can be applied to approximate unlearning methods in the literature. But this reviewer does not buy this claim. For example, Bourtoule et al., 2021 would trace back to the intermediate models before the data to be removed is in use. This approach could fail the proposed attack. o In machine unlearning approach in this paper, which is retraining from scratch, is also a cost-heavy way to do machine unlearning. Therefore, the comments on 288-290 on the strong assumptions or taking too much memory is not as convincing. o Typo: line 324, the second “learning” should be “unlearning”
Questions
See weakness.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Not applicable.
Summary
This paper studies poisoning attacks against machine unlearning systems, where the attacker injects a set of poisoning samples and a set of camouflage samples. When both sets of samples are present in the training set, the trained model performs similarly to the clean model but shows adversarial behaviors and misclassified a particular sample into a wrong label when the camouflage samples are deleted. Throurgh evaluations, the proposed attack can achieve reasonable poisoning success and the camouflaged samples also serve well in preserving correct predictions when present.
Strengths
1. The idea of attacking machine unlearning systems is very interesting. 2. The proposed attack, although majorly leverages the existing targeted poisoning attack, performs relatively effectively. 3. Evaluations show that the proposed attack may transfer across different model architectures and also unlearning methods.
Weaknesses
1. The extension to the practically relevant setting of misclassifying multiple test points is not demonstrated. 2. The attack requires a significant amount of poisoning points to work for more complex datasets such as ImageNet.
Questions
Overall, I think the problem studied in this paper is very interesting and the presentation of this paper is also very clear. However, I have the following concerns: 1. Misclassifying a single test point might not be practically relevant, attackers might be interested in investing a significant amount of poisoning points to misclassify as many test samples as possible. Although the authors discussed about the extension is straightforward, for practical implementation, I believe this will not work well. The original GradMatch attack performs poorly when the target number of samples exceeds maybe 5 images, not to mention that the proposed attack already performs inferior to the GradMatch attack in the standard training setting. So, I believe the authors may either tune down the statement slightly or demonstrate it through experiments (this will be ideal). 2. The authors mentioned that testing with different random seeds can lead to drastically different effectiveness and in practice, it is hard to know the random seed used by the victim (I hope I understand the paper correctly that the attacker and victim are indeed using different random seeds. Otherwise, the experiments probably be needed to rerun). In this regard, how can the attacker pick a good random seed to work well against the unknown seed by the victim? Repeating multiple times may not help as it is related to the unknown seed used by the victim. 3. The attack performs relatively poorly on more complex ImageNet datasets, and the performance of the GradMatch attack in the standard training is not listed. By checking the original paper, it seems the attack success of GradMatch is > 80% at <1% poisoning ratio, which means there is a big performance gap between the original GradMatch and the GradMatch in unlearning settings. This is probably a more worrying observation for me-- if there is an inherent barrier to poisoning complex models in unlearning settings, then we may have to worry too much about the poisoning threat. I guess this issue cannot be solved in a short amount of time but would be ideal if the authors can provide the attack performance when the total poisoning ratio is set as the common ratio used in prior poisoning literature (e.g., 1%). Related to this, the poisoning budget is usually 2x the budget of the poisoning standard setting, while in practice, attackers only have a limited budget. This implies the poisoning threat in machine unlearning will be less severe than poisoning in standard training. 4. There is an observation that the attack effectiveness varies drastically across model architectures. Do the authors have explanations for this? 5. When generating the camouflaged points, what is the intuition behind using the poisoned model $\theta_{cp}$ to generate the points? Is this a possible reason that leads to inferior performance of the attack on more complex datasets? ==================================================== This is to acknowledge that I have read the authors rebuttals carefully and my concerns are addressed.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
N/A
Summary
This paper proposes a new attack for security risks in the field of machine unlearning, namely the camouflaged data poisoning attack. The attacker adds poisoned and camouflaged points to the training data, and then triggers a deletion request to remove the camouflaged points, causing the model to misclassify the target points. The authors conduct an extensive evaluation to assess the performance of the proposed attack, and the results are promising.
Strengths
- Popular topics - Well-structured and easy to follow
Weaknesses
- Unrealistic assumptions - Needs more detailed explanations - Needs more complex datasets
Questions
The authors introduce a new attack surface in the context of machine unlearning called camouflaged data poisoning attacks. I like this topic because both data poisoning attacks and machine unlearning are two popular research topics. In addition, the authors clearly describe the threat model, methodology, and evaluation. I appreciate that. However, I do have the following concerns. - The authors design two algorithms for generating camouflage samples. The first one is through label flipping and the second one is through gradient matching. My concern is that the second scenario where an attacker has access to model gradients is unrealistic, especially in so-called image-based social media platforms. I do not believe that such a platform would have white-box access to the user. Although the authors discuss this point in the last section, I wish the authors would focus more on black-box settings. - Figure 1 provides the attack performance of the newly designed poisoning attack. Here, I wonder why the attack success rate on ResNet-50 is much lower than other model architectures. I hope the authors will have more discussion on this. I conjecture that one of the reasons behind this may be the much larger model complexity of ResNet-50. Therefore, I would further like the authors to add an ablation study on the impact of target model complexity. - Following the above, I am also curious about how the proposed attack performs on more complex datasets. Specifically, all experiments are conducted on 10-class datasets. However, in real-world scenarios, the task is likely to be more complex, with more than 10 classes. Therefore, I would like the authors to perform an investigation on more complex datasets, such as CIFAR-100 and ImageNet.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Insufficient experiments, conclusions may not generalize to complex architectures and datasets
Summary
This paper introduces a very interesting novel attack vector for tampering with the behavior of a machine learning model by influencing its training dataset. It poses that the practice of "unlearning" examples that are requested to be taken down can be exploited. The attack proceeds in several steps: 1. The adversary poisons the training data with two types of poisons: camouflages and regular poisons. 2. The adversary asks that the camouflages be unlearned (as might be their right under some legislation). 3. Unlearning is applied by the model developers, which triggers the poisoning effect and some examples are misclassified at inference time of the "unlearned" model.
Strengths
This is a strong and novel attack.
Weaknesses
I believe the biggest limitation of this work is that experiments are only conducted on small computer vision datasets. It would be very interesting to find out if these poisons work against modern text-to-image or large language models.
Questions
No questions from me.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
No concerns here.
Summary
This paper introduces a clean-label targeted data poisoning attack for machine unlearning pipelines. Gradient matching [A] is used to generate the poisons, and experiments use the retraining from scratch method for machine unlearning. The core idea is for an attacker to introduce poisons and camouflages to the clean dataset. The camouflages mask the effect of poisons, so the victim model trained on the combined dataset (clean + poison + camouflage) doesn't exhibit poisoned behavior. After this, the attacker makes an unlearning request to remove the camouflages. After the unlearning process, the model exhibits the poisoned behavior. The poisoning aims to misclassify a specific test sample to a chosen target class. [A] Jonas Geiping, et al. “Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching.” International Conference on Learning Representations. 2021.
Strengths
The paper proposes an original attack idea – a targeted poisoning attack in a machine unlearning setup. The optimization method used for generating the poisons (Gradient Matching) is a well-known method from poisoning literature to solve the bi-level problem. The experimental setup is decent with experiments covering a handful of datasets (CIFAR-10, ImageNette and Imagewoof) and machine learning models (SVMs, VGG, ResNets). I was intrigued by the question the paper poses, “We thus identify a new technical question of broader interest to the data poisoning community: Can one nullify a data poisoning attack by only adding points?” I agree that this direction of study might be of significant interest to the community.
Weaknesses
There are certain weaknesses in the claims of the paper, and the experiments are not thorough enough. They need improvement. 1. (Line 216) “camouflaging was successful ... provided that poisoning was successful. A camouflaged poisoning attack is successful if both poisoning and camouflaging were successful.” This explanation doesn't seem consistent with the numbers reported in the tables. If the camouflaging success rate is conditional on the poisoning being successful, I would think camouflaging success rate ≤ poisoning success rate. But that doesn't seem to be true in Row 3 of Table 1. Furthermore, if both these numbers are independent of each other, the tables should also report when both happen, as mentioned in the last line of the quoted text. 2. (Footnote) “We diverge slightly from the threat model described above, in that the adversary modifies rather than introduces new points. We do this for convenience and do not anticipate the results would qualitatively change.” I do not think this is true. If we have a clean image and also the (same clean image + perturbation) in the poisoning dataset, wouldn't it change the results? 3. (Amount of training – Line 244) “Each model was trained to have validation accuracy between 81-87% (depending on the architecture)”. 81-87% on CIFAR-10 seems to be quite a low number. It should be easy to achieve 90-95% on CIFAR-10 with VGG-11, VGG-16, Resnet-18, and Resnet-34. These models seem to be under trained. Given that no experiments are performed on ImageNet and compute shouldn't be a bottleneck for CIFAR-10 training, the models need to be trained more so that the experimental results have more value for future research. See https://github.com/kuangliu/pytorch-cifar for reference numbers. 4. (Extension to a set of targets) The attack threat model used here aims to target a single image from the test set. Though this is a starting point, the applicability of such a poison is quite limited. A natural extension would be to do this for a collection of targets. Though this is mentioned as future work in the discussion section, I would recommend that being a part of this current work.
Questions
There are some parts of the text which are not clear and will benefit with additional explanation. 1. (Line 172) What is $b_p$ and $b_c$? 2. (Line 173) What is $P$? 3. (Line 265) What is the validation accuracy for these models. I do not see them mentioned in the main text / appendix.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
A discussion on potential negative societal impact is missing. As this is a poisoning attack paper with implications on real-world deployments of machine learning models, I would recommend the authors to include a comment.
Thanks for the response
Thanks for the rebuttal. I think some of my concerns are addressed, while others are not likely to be solvable within a short amount of time and the academic resource budgets. I would encourage the authors to discuss the performance of attacking multiple points in the paper and also to make it clear in the paper that the attack performance variation results from the randomness on the attacker's side, not the victim's side. I wish there could be more insights on why the performance varies across architectures, but also agree that poisoning on large models is still complicated and would not penalize the authors for not addressing this clearly. I am raising my score accordingly.
Thank you for your consideration and response! Yes, we certainly will do so in the revised version of the paper.
As someone who has followed this field, I agree with the description by the authors of this attack as "clean label." Clean label attacks are those where the ground truth label of the image is not modified, as the authors describe here. For example, one of the first works to use the term clean label is Shafahi et al. 2018: https://arxiv.org/abs/1804.00792 It states in its abstract "The proposed attacks use 'clean-labels'; they don't require the attacker to have any control over the labeling of training data" In the case of this paper, the authors assert this is a clean label attack since an independent labeler looking at their images would label them as the "original" ground truth of the adversarial example.
Thank you, we appreciate Reviewer oSYx's confirmation on the definition of clean label. In the final version, we'll include a more precise description of what the term entails, as well an attribution to the work of Shafahi et al., 2018, as we too believe this is the first usage of this term.
Follow up!
Since we are approaching the end of the rebuttal period, please let us know if you have any further questions or concerns. We would love to engage with the reviewer if the reviewer has any pending concerns. Thank you, Authors
Follow up!
Since we are approaching the end of the rebuttal period, please let us know if you have any further questions or concerns. We would love to engage with the reviewer if the reviewer has any pending concerns. Thank you, Authors
Thank you for the detailed response. 1. It's now clear to me from the spelled out formulae of the metrics that the authors report conditional success of the camouflaging attack. It will be helpful to work these into the text either in words or as it is. Reviewer VjL7 raises a good point about low camouflaging performance for ResNet-50 (and MobileNetV2). Based on the metric definitions, a low poisoning success rate on ResNet-50 shouldn't influence camouflaging success rate. Even if one gets 25% poisoning rate, it's possible to get 100% camouflaging rate (Table 2, Row 1). Moreover, ResNet-50 gets 80% attack success rate, which is relatively high. (I am assuming Table 7 and Fig 4(left) report the same numbers.) It might be the case that on increasing model size, the attack efficacy decreases (as seen from the trend on ResNets (18, 34, 50)), which should be clarified. 2. I do not have a strong counterpoint to “However, we are unaware of any situation in the data poisoning literature where these two models qualitatively differ.” I'll need to think more about this, and it won't affect my final decision. 3. It will be great if the updated numbers with data augmentation training are added. 4. Would using a Bullseye Polytope objective improve camouflaging attack performance over Witches' Brew? As far as I know, it is designed to be effective for multiple targets. 5. Thank you for pointing out the locations where the symbols are defined. Additional Typo “Australian Terrier” (Fig 5 caption, Pg 20)
Thanks!
Thank you for your feedback. We will definitely add updated numbers with data augmentation training and clarify the definitions in the final version of the paper. After this is done, we should be able to confirm and discuss the effect of the model size on the attack success. **Incorporating Multiple Targets and comment about Bullseye Polytope"** From our understanding, the Bullseye Polytope procedure of Aghakhani, et al. 2021 considers a multi-target mode that generates poisons by using multiple images of the same target object (e.g. from different angles, positions, etc). Note that this is different from generating poisons for multiple (and independent) targets, which seems to be what the reviewer is requesting. We agree that extending camouflaged poisoning attacks, or even vanilla data poisoning attacks, to work for multiple independent targets is an interesting research direction. However, this is beyond the scope of our paper, but if resources permit, we would be happy to add more experiments for multi-targets with Bullseye Polytope in the final version of the paper. We also remark that we did run some experiments with Bullseye Polytope for generating our attack. We refer the reviewer to Appendix D for more details! We anticipate Bullseye Polytope to work better than gradient matching for multi-target settings since it generally requires way fewer poison images, e.g. 5-10 as compared to 200-500 for gradient matching.
Thank you for the discussion. The authors tried their best to address my concerns. I am willing to raise my score.
Follow up!
Since we are approaching the end of the rebuttal period, please let us know if you have any further questions or concerns. We would love to engage with the reviewer if the reviewer has any pending concerns. Thank you, Authors
Thank you for your response! > Our existing transfer experiments in Appendix C.6.3 address this concern... The transferability study does alleviate some of my concerns regarding the black-box scenario. However, as I previously noted, the performance difference between ResNet-50 and ResNet-18 caught my attention. I understand that this might be influenced by the success of the poisoning attack, but it prompts me to question the conditions under which transferability truly comes into play. > performed additional experiments on CIFAR-100... I appreciate the effort put into testing the attack on CIFAR100. Nevertheless, since it's quite similar to CIFAR10, its significance might be limited. It would be more impactful if similar experiments could be conducted on a larger and more diverse dataset like ImageNet. > Unfortunately, our group does not have access to the computational resources required to run experiments on ImageNet. The point about computational resources caught my attention. If launching the attack requires substantial resources even for the authors, does this suggest that it's even more infeasible for potential adversaries? In conclusion, while there are still some lingering concerns - I do believe experiments on larger datasets like Imagenet are necessary - I appreciate the authors' clarification regarding the transferability. I am raising my score from 4 to 5.
Response to computational resources for Imagenet
Thank you for your feedback. We want to clarify our comment that **" our group does not have access to the computational resources required to run experiments on ImageNet"**. We have been running our experiments on Google Cloud and found it practically impossible for us to upload the entire ImageNet dataset for our experiments. Thus, we resorted to smaller Imagenette and Imagewoof datasets (10 classes instead of 1000 classes). In terms of computation, the time taken by the adversary to generate camouflages and poisons via gradient matching for ImageNet should be smaller than the time taken to train a model from scratch. Thus, an adversary capable of training a model on Imagenet is likely to have the resources needed to implement the attack.
Decision
Accept (poster)