Summary
The paper proposes a novel detection and defense method for backdoored models. The new method is motivated by the finding that existing state-of-the-art trigger-inversion methods, like BTI-DBF, rely on strong backdoor features, which might not always be present, such as for BadNet-type triggers. As a solution, the paper proposes BAN (Backdoors activated by Adversarial neuron Noise), a novel detection method based on the findings that models with backdoors are more sensitive to adversarial noise than benign ones, allowing the identification of neurons responsible for the backdoor function. Practically, BAN computes neuron-specific noise inspired by adversarial attacks like PGD to maximize the model's classification loss on clean and unseen data. The method tries to decouple benign and backdoor features by optimizing a sparse neuron mask on the loss behavior of the model. Intuitively, a backdoored model tends to predict the target class label for clean inputs and the adversarial perturbed neurons, whereas a benign model shows fewer misclassifications under comparable neuron noise. Evaluated against common backdoor attacks and defense methods, BAN demonstrates improvements to existing approaches and robustness against adaptive attacks.
Strengths
- The proposed defense method adds an interesting aspect to existing backdoor defense methods by improving the feature decoupling between benign and backdoor features. Whereas the decoupling optimization is conceptionally simple (which is not bad at all), it demonstrates strong results on the evaluation benchmarks and beats existing detection and defense methods markedly.
- The paper is well-written and easy to follow. Most parts of the proposed method is sufficiently placed in existing research, and the open research problems and the proposed solution are clearly presented and described. I enjoyed reading the paper.
- The evaluation is comprehensive and compares the proposed method against numerous attacks and defenses. It also includes the all-to-all setting, which is often ignored in literature. Evaluations of adaptive attacks underline the effectiveness and robustness of the approach.
- Empirical solutions and theoretical considerations sufficiently support all claims in the paper.
Weaknesses
- The evaluation focuses on common (comparably shallow) CNN architectures. However, given that ViT also plays an increasing role in image classification, showing the effectiveness of the approach on ViT architectures in contrast to traditional CNNs would further support the efficiency claims of the method.
- The evaluation further focuses on dirty label attacks, i.e., attacks that change the label of a training example. However, the method's efficiency on clean-label attacks is not demonstrated. Including 1-2 clean label attacks in the evaluation would further strengthen the results.
- Some contributions of the paper should be stated more clearly and set apart from existing approaches in literature. For example, the feature decoupling process described in 3.3 is quite similar to the method by Xu et al. [1], and it is unclear to me (by only reading this paper) what exactly distinguishes the proposed method from the one in the existing literature.
Small remarks:
- Some captions are missing details. For example, the caption of Table 5 does not clearly state the investigated setting. Which dataset and model architecture are used here?
- The font in the figures could be increased. Some legends and texts are hard to read without zooming in.
- There is a typo in line 142: "experiemnts" -> experiments.
- Table 7 seems like a sensitivity analysis (measuring the sensitivity of the method to the hyperparameter selection) instead of an ablation study (impact of deactivating certain parts of the method).
[1] Xu et al. "Towards Reliable and Efficient Backdoor Trigger Inversion via Decoupling Benign Features". ICLR 2024
Questions
- L303: "FeatureRE is designed for all-to-one attacks, so we use target label 0 here for FeatureRE." -> Does it make sense to include an approach designed for all-to-one attacks in an all-to-all setting? The comparison here with the other approaches seems misleading.
- Table 4: What are the BA (benign accuracy) and ASR (attack success rate) of models trained only on clean data without any poisoned data samples? It would be interesting to compare the defended backdoored models to a clean model. Since ImageNet (and its subsets) are complex datasets to learn, there will also be misclassifications that might, by chance, lead to the prediction of target class labels (even without any backdoor integration). Given ASR of the defended models, it would be interesting to see a baseline ASR (achieved only by random misclassifications) to assess if the remaining ASR after applying the defense methods is due to remaining backdoor behavior or simple random model behavior.
- The proposed method is based on the assumption that under neuron noise, a backdoored model tends to predict the target class label. In contrast, the predicted class labels for the benign model are rather equally distributed. This makes sense for datasets with clearly distinguishable classes, e.g., CIFAR10. Let us assume that there exist two classes in the dataset that are semantically very similar, e.g., two visually similar (but still different) dog breeds among other clearly separable classes (bug, plane, etc.). Let us further assume that the backdoor target is to change the label from the one dog class to the other one. If we now apply the neuron noise approach to a benign and a backdoored model, can we still assume that only the backdoored model will tend to predict the other dog class (the target class) under neuron noise? Or will the benign model under neuron noise behave similarly, since the other dog class is probably close in the feature space, and adding noise to the neuron activations could lead to predictions for the second dog class (which happens to be also the backdoor target)? Phrased differently, can we assume the proposed BAN method also works reliably on datasets with semantically similar classes, if the backdoor targets label changes from one class to the other?
Limitations
Limitations are sufficiently discussed in Appx. A. Since the paper proposes a novel defense method for backdoor attacks, no negative societal impact is expected.