Backdoor attacks aim to inject a backdoor into a classifier such that it predicts any input with an attacker-chosen backdoor trigger as an attacker-chosen target class. Existing backdoor attacks require either retraining the classifier with some clean data or modifying the model's architecture. As a result, they are 1) not applicable when clean data is unavailable, 2) less efficient when the model is large, and 3) less stealthy due to architecture changes. In this work, we propose DFBA, a novel retraining-free and data-free backdoor attack without changing the model architecture. Technically, our proposed method modifies a few parameters of a classifier to inject a backdoor. Through theoretical analysis, we verify that our injected backdoor is provably undetectable and unremovable by various state-of-the-art defenses under mild assumptions. Our evaluation on multiple datasets further demonstrates that our injected backdoor: 1) incurs negligible classification loss, 2) achieves 100% attack success rates, and 3) bypasses six existing state-of-the-art defenses. Moreover, our comparison with a state-of-the-art non-data-free backdoor attack shows our attack is more stealthy and effective against various defenses while achieving less classification accuracy loss.
Paper
Similar papers
Peer review
Summary
This work introduces data free backdoor attacks (DFBA). The idea behind this attack is to introduce a backdoor into already trained neural nets by manually modifying parameter weights without requiring fine-tuning or any initial clean data. The backdoor is implemented by manually defining a path from the input of the DNN to the target class' softmax output that will override all other class activations via magnitude if a pre-defined trigger is present in the input, thereby classifying the input as the target class. The backdoor trigger is calculated from the current parameter weights of the chosen first layer backdoor neuron. Experiments reported that DFBA achieved a 100\% ASR in all experiments, while it only hurt clean accuracy by a few percentage points.
Strengths
* (Major) The presented attack, DFBA, is easy to grasp, intuitively makes sense that it works, has good performance, and can be injected quickly without a dataset or fine-tuning. Overall, the idea seems strong. * (Moderate) The topic this work tackles, backdoor attacks, are becoming increasingly important to investigate. This work outlines a possibly new attack vector that should receive attention. * (Moderate) Sec 4.1 (Technical Details) is overall well written and understandable. I especially appreciate the extra text dedicated to interpreting the meaning of the math under Lemma 1.
Weaknesses
* (Major) The claims that this defense is undetectable and unremovable are predicated on the assumption that the defender is not informed about this attack and is not looking for anything like it. Intuitively, it would not seem hard to detect several neurons (one in each layer) that have 0 weights for all but the preceding one-hot neuron if they were being looked for. For this reason, it may be good to tone down these claims or at least caveat them with the assumption the defender is unaware the attack exists and only defenses created to defend against other types of attacks are being used. * (Major) I did not see details on how many features were required to be part of the trigger, and how this choice was ensured to be stealthy, and a fair comparison to baseline backdoor methods. * (Moderate) The theoretical analysis assumes that no clean samples will activate the backdoor, and that the defender could not find the backdoor. However, the experimental results did show that one of these assumptions can be empirically broken, in that at least one clean image activated the backdoor, and therefore, a defender may be able to find the backdoor path just by using some (or many) clean samples. * (Moderate) The results for whether DFBA is effective against state-of-the-art defenses are not in the main text, even though it is one of the four main contributions presented in the introduction. To be claimed as a major contribution, at least an overview of the results should be presented in the main text. * (Moderate) As I understand it, this method is limited to being used in image classifiers with a DNN architecture that uses convolution neural nets and/or dense nets, which are only connected by ReLU activations. The attack must also be catered specifically to the architecture being used. This limitation should be made known in the main text somewhere. * (Minor) In Sec 3.3, the efficiency goal is to make an attack that is efficient, which seems under-defined. This goal could use some quantifiable metric to verify that it is efficient in Sec 3.3. * (Minor) There are a few grammar and syntax errors throughout the paper, and it would benefit from a thorough read-through.
Questions
* The theoretical analysis assumes that removing L-1 neurons removed (where L is the number of layers in a DNN) will have negligible clean accuracy effects. However, it does seem, according to Table 1, that backdoor accuracy can be non-trivially affected by removing these neurons. How does this affect the claims of the theoretical analysis? * What are the details on how many features are used in a typical trigger? Was this constrained in some way? How do the triggers compare with the triggers used in other baseline backdoor methods and how can they be fairly compared? * How would an attacker using DFBA adapt to a defender knowing about this attack? Could this analysis be taken into account? * Is one potential defense simply to use GeLU (or some other non-ReLU) activation instead of ReLU? Could DFBA be adapted to still work on this and still be resistant to fine-tuning defenses?
Rating
6
Confidence
4
Soundness
2
Presentation
4
Contribution
3
Limitations
As I understand it, this method is limited to being used in image classifiers with a DNN architecture that uses convolution neural nets and/or dense nets, which are only connected by ReLU activations. The attack must also be catered specifically to the architecture being used. This limitation should be made known in the main text somewhere.
Summary
In this paper, the authors propose DFBA, a novel approach for injecting backdoors into pre-trained classifiers without the need for retraining or access to clean data. This method stands out by not altering the model's architecture, which enhances its stealthiness and efficiency. The authors claim that DFBA's backdoor is undetectable and unremovable by state-of-the-art defenses under mild assumptions. Empirical evaluations on various datasets demonstrate the attack's effectiveness.
Strengths
1. This paper is well-written. 2. The experimental results demonstrate that the proposed attack is capable of evading state-of-the-art backdoor defense mechanisms.
Weaknesses
1. The process of "Neuron Selection for a CNN" is not clearly described in the paper. The authors need to provide more details as well as the distinctions from FCN. 2. This paper does not pioneer the concept of a data-free backdoor, as it was introduced after [1]. Therefore, I believe the authors need to reconsider the title of the paper and the name of the proposed method. Moreover, the authors need to provide a detailed discussion and comparison with [1] within the paper, rather than simply opting for a previous parameter-modification-based backdoor method, which gives me the impression of an insufficient evaluation. 3. I appreciate the authors' efforts in the paper to verify the proposed method's resilience against existing defense mechanisms, but I believe the evaluation of the attack is inadequate. For instance, assessments should be conducted across a broader range of datasets and models, as well as considering various triggers and multiple target classes for the attack. I am skeptical about the reported 100% ASR values in Table 1, as this is uncommon among current mainstream backdoor works. The authors need to provide more explanations and evidence for these experimental results. 4. The technical details and parameter settings provided in the paper are insufficient for reproducing the experimental results presented. The authors need to supply the code to ensure reproducibility. 5. The experiments in this paper do not provide error bars or results from experiments with different random seeds, which raises my concerns about the validity and stability of the experimental outcomes. Overall, I greatly appreciate the work presented in this paper, but the authors need to provide more discussion and comparison with [1], which is a key condition affecting the paper's potential for acceptance. I would be willing to raise my score after the authors address my concerns. References [1] Lv, P., Yue, C., Liang, R., Yang, Y., Zhang, S., Ma, H., & Chen, K. (2023). A data-free backdoor injection approach in neural networks. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 2671-2688).
Questions
As the most relevant work [1] mentioned, "Our approach is generic, capable of injecting backdoors into various tasks and models, e.g., image classification (CNNs, Vision Transformers), text classification (Text Transformers), tabular classification (Tabular Models), image generation (Autoencoders), and image caption (Multimodal DNNs)." How does the method proposed in this paper perform under these settings?
Rating
5
Confidence
5
Soundness
2
Presentation
3
Contribution
3
Limitations
The method proposed in this paper has not been effectively validated for its efficacy across different downstream tasks and a broader range of model architectures, such as Vision Transformers, Text Transformers, Tabular Models, Autoencoders, and Multimodal DNNs. Additionally, the concept of a data-free backdoor introduced in this paper is not the first of its kind.
Summary
In this work, the authors design DFBA, a novel retraining-free and data-free backdoor attack that does not alter the architecture of a pre-trained classifier. They theoretically prove that DFBA can evade multiple state-of-the-art defenses under mild assumptions. Their evaluation on various datasets demonstrates that DFBA is more effective than existing attacks in terms of attack efficacy and utility maintenance. Additionally, they evaluate DFBA's effectiveness against multiple state-of-the-art defenses, showing that these defenses cannot counter their attack. Ablation studies further demonstrate that DFBA is insensitive to hyperparameter changes.
Strengths
1. This paper introduces a novel backdoor attack that does not require retraining, data, or changes to the model architecture. It also provides theoretical analysis to prove the effectiveness of the proposed method. 2. The authors consider several advanced backdoor defense methods and demonstrate that the proposed DFBA can partially overcome these defenses.
Weaknesses
1. For FCN, I understand how DFBA works. However, for networks like VGG and ResNet, which include convolutional layers and BN layers, I am not entirely clear on how DFBA functions (even though it is mentioned in the appendix). I hope the authors can clarify this further and provide open-source code. 2. I found that the proposed method is quite similar to the approach in "Backdoor Attack for Federated Learning with Fake Clients." Although that work focuses on the federated learning scenario, the method of injecting the backdoor is almost identical to that in this paper. I hope the authors can explain this. 3. The authors could compare more data-free backdoor methods to highlight their method's superiority, such as "A Data-Free Backdoor Injection Approach in Neural Networks." 4. Since there is not much work on data-free backdoor attacks in centralized training scenarios, I believe that related work on data-free backdoor attacks in distributed training scenarios should be included in the related work section. Examples include "Backdoor Attack for Federated Learning with Fake Clients" and "DarkFed: A Data-Free Backdoor Attack in Federated Learning." 5. Typos: A summation symbol is missing in line 226, there are two "i.e." in line 303, and the two "xi" in line 314 are inconsistent.
Questions
My questions are included in weaknesses.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The limitations are included in weaknesses.
Summary
This paper proposes a backdoor attack that directly modifies the model parameters and does not rely on any data. By designing a backdoor switch in the first layer, optimizing the trigger, and amplifying outputs in the following layers, the method creates a backdoor path that can be activated by backdoored input and does not respond to clean input. Experiments show the method can achieve high attack success rates while having low clean accuracy loss and being resilient to several current state-of-the-art defenses.
Strengths
1. This method does not need any data and does not need to modify the model’s architecture, which is efficient and highly applicable. 2. This method shows high attack success rates and high clean accuracies and can bypass current defenses, which is a good direction to be studied. 3. The method generally makes sense and is clear.
Weaknesses
1. My concerns are mainly about the writing. a) Since “our DFBA is effective under state-of-the-art“ is one of the main claims, at least one table should be presented in the main paper, instead of in the supplementary. b) Some sentences are repeated and should include more details instead, e.g. line 245, line 245, line 369-372, etc. For example, the paragraph “Our DFBA is efficient“ needs not to state the method “directly change the parameters” and “is efficient“ repeatedly, but needs to include more specific contents, e.g. comparing to other methods. c) Some terms and definitions can be switched to improve readability further. For example, $x=[x_1,x_2,\cdots,x_d]\in \mathbb R^d$ where d represents the number of pixels is not very common in the computer vision field; the terminology “neuron” and “feature map” are somewhat less common in the CNN than filter and channel. 2. The stability of this method is not stated, making the results less convincing.
Questions
1. The stability, as mentioned in line 352 “thus randomly selection neurons are more likely to impact classification accuracy”, how would the randomness impact this method? I am curious how much difference could be caused especially when different neurons in the first layer are chosen, and also the impact of randomness on different model architectures. 2. There are some other pruning-based methods such as ANP [1] and RNP [2], is the method effective for those either? [1] Reconstructive neuron pruning for backdoor defense. [2] Adversarial neuron pruning purifies backdoored deep models.
Rating
6
Confidence
3
Soundness
4
Presentation
2
Contribution
3
Limitations
The authors discuss the limitations.
Summary
This paper proposes a strategy for injecting backdoors into a DNN without the attacker requiring access to the training data of the model or having to change the architecture of the model. The attack is executed by directly manipulating the parameters of the neural network. Concretely, this is achieved by selecting a backdoor path, consisting of a single neuron across every layer of the network, which are maximally activated for the backdoor trigger but remains inactive for clean inputs. In this way, the backdoor has a minimal impact on the clean accuracy of the model. The authors also provide a theoretical analysis of the utility, efficiency and robustness of the attack against state of the art defenses.
Strengths
* The attack does not require access to the original training data for the model for injecting the backdoor trigger. * Does not require architectural modifications to the model. * The attack is shown to be effective in maintaining the backdoor accuracy close to clean accuracy for most of the datasets.
Weaknesses
* It is not clear how the attack can be extended to include multiple backdoor triggers into the same model and how that would impact the backdoor accuracy of the model. * Would this attack be effective on large models with billions of parameters? How does one go about choosing the backdoor path in large models? Given on ImageNet we already see that there is around 3% drop between CA and BA, would this attack scale to larger and more complex datasets. * It is unclear if the defenses were adapted to the backdoor attack before evaluation? The strength of an attack should usually be evaluated against defenses that are modified to make them aware of the attack.
Questions
* How difficult is it to port the attack to architectures different from FFN and ConvNets? * How does the attack scale with model and data sizes?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations are mentioned.
Looking forward to authors-reviewers discussions
Dear Reviewer AzFZ, We sincerely appreciate the time and effort you have invested in reviewing our paper! We believe we have thoroughly addressed all your concerns, particularly your suggestions regarding multiple backdoor paths and your concerns about adaptive attacks. We have conducted experiments to validate these points and obtained positive results. As the discussion period for NeurIPS 2024 is nearing its end, we would genuinely like to know if you find our rebuttal helpful, and we are more than willing to address any remaining questions you may have. Thanks, Authors
Thank you for a detailed response. Appreciate it. The adaptive defenses in CA2 are interesting and the initial results look promising. I am not fully convinced with the responses of Q3 and Q5 above, especially about the generalizability of the attack. However, I do feel these are good potential directions for extension of this work. For now, I will maintain my score.
We sincerely thank the reviewer for their service and for maintaining a positive attitude towards our paper! Below, we would like to share further clarifications on Q3 and Q5: Regarding Q3: We apologize for any confusion. In the last few days of the response period, we are still trying to obtain some more empirical evidence to show you that further scaling of the model size would not lead to significant drop in terms of the gap between CA and BA. In particular, for the ImageNet data, we further conduct experiments on the Resnet 152 model | Model | CA | BA | |:------------------|:-----:|------:| | ResNet-50 | 76.13 | 73.51 | | ResNet-101 | 77.38 | 74.70 | | ResNet-152 | 78.33 | 75.27 | This showcases that further increasing the model size would not enlarge the CA/BA gap significantly and our method can still work on those large models. Hope this further clears your concern on the larger models. Regarding Q5: Since our method directly modifies the model weights without any data, it needs specific designs for specific architectures. Yet as we mentioned in the previous response: as long as we can isolate a certain path in the model, we can deploy our DFBA with theoretical guarantees. We are sorry that given the short response time, we cannot quickly adapt DFBA to other architectures and give you further results but we will keep that as our future work. Nevertheless, DFBA can achieve nearly 100% ASR in less than 1 second of injection time, with no data and minimal impact on the model's original performance. We believe this somewhat compensates for the current limitations of DFBA in terms of extensibility. Once again, we thank the reviewer for their profound insights and productive discussion, as well as for their support and recognition of our paper!
Many thanks!
We sincerely appreciate the reviewer's insights and are thankful for the increased score post-rebuttal. It truly motivates our ongoing research efforts!
Thank you to the authors for their thorough responses, which addressed the majority of my concerns. I will raise my score.
Thank you！
Thank you for taking the time to re-evaluate our paper after the rebuttal. We're grateful for your constructive feedback and the positive shift in assessment!
Response from the reviewer
Thank you for your response. However, I still have two major concerns that haven't been addressed. 1) Regarding Q4: The paper presented at USENIX Security '23 provides extensive validation results across various architectures, datasets, and tasks. As a direct comparison, this work should be evaluated under similarly broad settings. 2) Regarding Q5: I am skeptical about the 100% ASR result, which may be due to the lack of validation across more diverse settings. I look forward to your further response to determine my final score.
Authors' Responses to The Reviewer's Follow-up Questions
We greatly appreciate the reviewer for further engagement in the discussions! We hope the following responses can further clarify the reviewer’s follow-up questions. We will first clarify your Q5 and then Q4. **Regarding Q5**: We are sorry for the confusion. We would like to first clarify that the 100% ASR is not due to the lack of experiments in more diverse settings but the choice of hyperparameter $\gamma$. In simple terms, we directly modify the model to create a new backdoor path that allows the target class to achieve any confidence score we want (the key is to control the amplification factor $\gamma$). For example, with $\gamma = 1000$ and other parameters at default values, the logit value on the target class when the backdoor is activated in the ResNet18 model will be approximately $1e56$, far greater than any normal logit values. Thus it can easily surpass any other classes and make sure that when the trigger presents and activated, the output is always the target class and achieve 100% ASR. However, a super large $\gamma$ may easily get detected so we cannot use an arbitrarily large $\gamma$ here. When the amplification factor $\gamma$ is small, the ASR may drop below 100%. As shown in our ablation (Figure 7.b), for the CNN model, when $\gamma = 5$, the ASR is about 75%, and when $\gamma = 5.5$, the ASR is about 95%. Generally, a slightly larger $\gamma$ can achieve an ASR around 100%. Thus under our hyperparameter settings, we empirically observed 100% ASR for all our main experiments. **Regarding Q4**: We understand your concern on comparing with Lv et al. [1] for more various experimental settings. However, we would like to first emphasize our key difference with [1]. Please note that the core idea of Lv et al. [1]'s method is to fine-tune various models using a **substitute dataset** with triggers. Therefore, it can naturally and easily be applied to any model architecture or standard tasks. However, it relies on the availability and quality of such “substitute dataset”. While our method **directly injects backdoors by modifying the model parameters** with no need of any type of “substitute data”. In this sense, our DFBA is also "substitutive data-free". Therefore, it is a bit unfair for us to compare with Lv et al. [1] on various experimental settings: holding the additional substitute dataset means all they need to do is to change the substitute dataset and fine-tune the model. For us, it requires specific designs for specific model structures/tasks. Despite that, in our current experiment (see CQ3), we still achieve better attack performances compared with Lv et al. [1]. Nevertheless, we are trying our best to provide you with some additional experimental results to showcase that our method is widely applicable on various tasks: We have added an experiment on a different malware detection task. We used the DikeDataset as the benign dataset and the Malimg dataset (25 classes) as the malware dataset to train a model to distinguish between benign and malicious software and determine the specific type of malware. We trained ResNet-18 on this dataset 20 times with a learning rate of 0.01. Then, we injected a backdoor using the default parameters described in the paper. The results show a clean accuracy of 97.64%, backdoor accuracy of 97.04%, and ASR of 100%. This demonstrates the effectiveness of our method across different task types. We will include more experiments on more diverse tasks in the final revision. Thank you very much! We hope these explanations address your concerns, and we look forward to your further response!
Final response from the reviewer
Thank you for your response! I greatly appreciate the new efforts made in this paper to implement data-free backdoor techniques. However, I still believe that the current version's experimental evaluation of the attack across different backbone networks, datasets, and tasks is insufficient compared to USENIX Security '23 . Overall, I maintain a positive view of this paper, so I will keep my score. Good Luck!
Thank you!
Thank you for your prompt reply! We understand the reviewer's concerns, and we are currently making our utmost efforts to provide additional results. We greatly appreciate your insightful comments and the productive discussion we've had. We also thank you for acknowledging our contributions!
The authors have addressed all the concerns well and have also shown their understanding of backdoor defenses. Therefore, I would like to raise my score to 6.
Thank you!
We sincerely appreciate the reviewer's thoughtful reconsideration of our paper following the rebuttal. Thank you for recognizing the improvements and adjustments we made!
Decision
Accept (poster)