From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models

Summary

The paper proposes a new poisoning attack for diffusion models (DMs). While previous work tried to poison/backdoor DMs by altering the training process or the optimization objective, the paper proposes a poisoning attack by only altering the training data. To poison DMs, a trigger is inserted into training images, and the labels of the poisoned samples are changed to the target class. The resulting DM, trained on this poisoned dataset, generates images not aligned with the given prompt or images containing the trigger pattern used for poisoning. Based on this behavior, insights are presented that might help protect DMs against poisoning attacks, and a different view on data replication in DMs is given.

Strengths

- The paper tackles a very important topic as the risk of poisoned data is increasing when training DMs on publicly available data scraped from the web - The insight that DMs generate images of the target class with the trigger, even though the trigger has not been present in the target class training images, is very intriguing. However, the paper doesn't really give an intuition or explanation on why this is the case (see questions).

Weaknesses

- Only training details about the Caltech15 dataset are provided in the appendix. (see questions) - It is unclear how this proposed method can be applied to datasets like LAION or other uncurated/unstructured datasets without clearly separated classes. - In the experimental setting, it is stated that experiments on CIFAR-10 are conducted. However, in the experimental evaluation, there are results for CIFAR-10. Only ImageNette and Caltech15 are used to show the effectiveness of the poisoning attack. (see questions) - The paper is sometimes hard to read, and in parts, it is difficult to grasp what the authors want to convey as the take-away message of the paper is not really clear, in my opinion. - Using the "poisoned DM" as a defense against poisoning attacks is not very realistic or applicable, in my opinion. In reality, a DM would first have to be trained to generate data and apply the poisoning detection method to the generated data before even starting to train the classifier. In addition, the improvement of the AUROC for the poisoning detection methods is only very minor (in most cases, less than 1 percentage point improvement of the AUROC value). - The data replication experiments are not really meaningful, in my opinion. If we look at replicated images, it is expected that these images are replicated more than randomly chosen images. The experiment would be more meaningful if the same images would be once poisoned and once not poisoned. This would give insight into whether the poisoning really affects the data replication abilities of the DM. - there are two other works [1, 2] that use DMs for defending against poisoning attacks that should be mentioned in the related work part [1] Zhou et al., DataElixir: Purifying Poisoned Dataset to Mitigate Backdoor Attacks via Diffusion Models, AAAI 2024 [2] Struppek et al., Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data, NeurIPS 2023 Workshop BUGS Misc: - Many of the cited papers are arXiv papers and not the conference versions (VillanDiffusion is NeurIPS, "Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning" is "conference on multimedia", Rickrolling the artist is NeurIPS, etc.). Please cite the proper conference versions of the papers. - The titles in the references only include lower characters. This seems to be a bibtex/latex problem. - Reading "the work [...]" is not really smooth. Instead, it would be better to write the author's names as "Chou et al. have done ...." - The links in the appendix should be blue to indicate that they are clickable. I almost missed them. Also, it might be preferable to show the URL so the reader knows which site is linked without clicking on the link in the first place. The paper tackles a very interesting problem, and the discovered phenomena seem to be very surprising. However, in my opinion, the paper is not quite ready for publication because of the unclear take-away message and the sometimes hard-to-read text.

Questions

**Q1:** How many samples were used to train the DMs on the ImageNette and the CIFAR-10 dataset? **Q2:** What are the experimental results for CIFAR-10? **Q3:** Why choose the black and white square as the first trigger? Why not just use a uni-colored square as in the original BadNets paper? **Q4:** I can imagine that the appearance of the trigger also plays a significant role in whether the poisoning is successful or not. You have chosen the black and white square. Does the same phenomenon also appear with other patterns? **Q5:** How many samples were used to calculate the FID scores in Table 2? **Q6:** What is the reasoning/intuition behind the phenomenon that the DMs seem to generate an image of the target class containing the trigger, even though the target class images in the training set didn't have the triggers?

Rating

6

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

Some of the work's limitations are addressed. However, I think it is important to discuss how realistic it is that the proposed poisoning method is used in such a realistic scenario and not only to investigate the behavior of DMs on poisoned data in an artificial setting. Additionally, the limitations of the proposed defense method should be discussed.

Summary

The paper investigates the impact of BadNets-like data poisoning attacks on state-of-the-art diffusion models (DMs) used for image generation. Unlike previous studies that required modifications to the diffusion training and sampling procedures, this work examines the effects of poisoning the training dataset alone. The study uncovers dual effects of data poisoning, which not only degrade the generative performance of DMs but also provide defensive advantages for image classification tasks. Key findings include the misalignment between input prompts and generated images, the amplification of trigger generations, and the linkage between data poisoning and data replications. The major contributions of this paper are as follows. It demonstrates that diffusion models (DMs) are vulnerable to BadNets-like data poisoning attacks, leading to two significant adverse effects: (1) misalignment between input prompts and generated images, and (2) an increased generation of images with embedded triggers, referred to as 'trigger amplification'. The study identifies a phase transition in the poisoning effect relative to the poisoning ratio, revealing the nuanced dynamics of data poisoning in DMs. The proposed 'Castle Walls' concept introduces defensive strategies for image classification, including leveraging trigger amplification for detecting poisoned training data, training classifiers with images from poisoned DMs before the phase transition to mitigate poisoning, and using DMs as image classifiers to enhance robustness against attacks. Additionally, the paper establishes a connection between data poisoning and data replication in DMs, showing that introducing triggers into replicated training data exacerbates both the replication problem and the impact of poisoning, thus highlighting the inherent data memorization tendencies of DMs.

Strengths

Originality: The paper presents an innovative investigation into the impact of BadNets-like data poisoning attacks on state-of-the-art diffusion models (DMs) used for image generation. Unlike previous studies that require modifications to the diffusion training and sampling procedures, this work uniquely focuses on the effects of poisoning the training dataset alone. This fresh perspective uncovers dual effects of data poisoning, revealing both degradation in generative performance and potential defensive advantages for image classification tasks. The introduction of the 'Castle Walls' concept for defensive strategies is original, offering new ways to leverage data poisoning effects to enhance robustness against attacks. Quality: The quality of the research is reflected in its comprehensive experimental analysis and the depth of its findings. The study methodically demonstrates the vulnerability of DMs to BadNets-like attacks, detailing how these attacks cause misalignment between input prompts and generated images and amplify trigger generations. The paper includes a thorough examination of defensive strategies, including the innovative use of poisoned DMs for training classifiers. Clarity: The paper is well-structured and clearly communicates its methodology, findings, and implications. The key concepts and contributions are articulated in an accessible manner, with detailed explanations of the experimental setup and results. While there are minor editorial issues, such as the need for clarification in figure captions and consistent notation, these do not significantly detract from the overall clarity of the paper. The inclusion of detailed figures and tables aids in the clear presentation of the data and results. Significance: The significance of this work lies in its potential to substantially enhance the understanding and robustness of DMs in the face of data poisoning attacks. By uncovering the dual effects of data poisoning and proposing innovative defensive strategies, the paper provides valuable insights that can inform future research and practical applications. The connection established between data poisoning and data replication highlights the inherent data memorization tendencies of DMs, offering a deeper understanding of their vulnerabilities.

Weaknesses

Additional statistical analysis (e.g., confidence intervals) could strengthen the findings by accounting for variability and ensuring the observed improvements are statistically significant. Experimental Robustness: The lack of reported error bars due to computational expense raises concerns about the robustness and representativeness of the experimental results. Without statistical measures of variability, it is challenging to assess the reliability of the findings. Constructive suggestion: Provide some supporting evidence or alternative measures to demonstrate the robustness of the results, such as reporting confidence intervals for a subset of the experiments. Comprehensive Defensive Strategies: While the 'Castle Walls' concept is innovative, the practical implementation details of these defensive strategies are not fully explored. Constructive suggestion: Provide more detailed guidelines and examples on how these strategies can be implemented in real-world scenarios to enhance their practical applicability.

Questions

In the figure captions, there is mention of G3 and G4 (that do not contain trigger), but these are not referred to in Figure 2 itself (only G1 and G2 are). Highlight in the text why these are missing and now shown? Checklist - Q7 Justification: Error bars are not reported because it would be too computationally expensive. How can we have confidence that the experimental results are representative and robust and not prone to statistical chance. Provide some supporting evidence. When non-monotic results are observed (for example Bad-Nets 2 on ImageNette, SD, Caltech15), explain why increasing the poisoning rate from 1 to 5% provides an AUROC improvement but an increase from 5% to 10%. Line 217, Page 6, Use the same notation as in the paper. “Fig A3 presents” -> A3 of which figure? Provide full reference.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed key aspects of their work, but several limitations require further attention to strengthen the paper. Experimental Robustness: The lack of error bars due to computational expense raises concerns about the robustness of the results. Without statistical validation, it is difficult to ensure the findings are consistent. Constructive suggestion: Include confidence intervals or statistical validation for a subset of experiments to enhance result reliability. Practical Implementation of Defensive Strategies: The 'Castle Walls' concept introduces novel defenses, but practical implementation details are lacking. Constructive suggestion: Provide detailed guidelines and examples for implementing these defensive strategies in real-world scenarios. Broader Societal Impact: The potential negative societal impacts of data poisoning are not thoroughly discussed in the main paper. Constructive suggestion: Discuss the broader societal implications and ethical considerations of your findings, including potential misuse and guidelines to mitigate negative impacts.

Summary

This paper investigates backdoor attacks against diffusion models. Unlike previous works that require both injecting poisoned data samples and manipulating the training loss function, this study focuses solely on poisoning training data samples during the training phase. The research demonstrates that backdoor attacks not only compromise the functionality of diffusion models (resulting in incorrect images misaligned with the intended text conditions) but also amplify the presence of triggers, a phenomenon termed 'trigger amplification.' This trigger amplification can be utilized to enhance the detection of poisoned training data, thereby providing a defensive advantage.

Strengths

-- This paper is easy to follow. -- This paper demonstrates that simply poisoning the training dataset can effectively backdoor diffusion models. -- Conduct comprehensive experiments. Impressive results especially in attack success rate. -- Discuss the limitation of the proposed attack and future work.

Weaknesses

-- The evaluation of the proposed attacks is limited to 3 datasets: CIFAR10, ImageNette and Caltech15.

Questions

1. It would be better if the authors can evaluate the proposed method on more datasets such as ImageNet1K and CIFAR-100 2. It is suggested that the attack model be described in a separate section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors addressed the limitations.

Summary

The paper studies BadNet-like poisoning attacks in diffusion models from both attack and defense perspectives.

Strengths

1. I think the paper makes interesting observations for the community, especially regarding the phenomenon of trigger amplification. 2. The evaluation seems quite comprehensive, considering multiple datasets, models, attacks, and detection methods.

Weaknesses

1. Even though the authors consider many settings, the experiments are run only once (no error bars are shown). 2. While in Table 4, the attacks' success rates are reduced when the poison percentage is up to 5%, I am wondering if they are amplified for higher poison percentages. If so, how could the defender use this as a defense in practice if they do not have any knowledge about the poison percentage? 3. The paper is fully empirical. Minor comment: at line 252 "comapred" should be "compared".

Questions

See weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Response Rebuttal

Thank you for your detailed answer. I appreciate your detailed rebuttal and the additional insights. **W1 & Q1:** Thank you, this answers my question. **W2:** Thank you for your answer. The results on LAION seem very promising. **W3 & Q2:** Thank you for pointing out where to find the CIFAR-10 results. **W4 &W8:** I think it was a bit difficult for me to find the additional results in the appendix as it was not clearly stated in the paper (e.g., that the CIFAR-10 experiments are in the appendix). **W5 & Limit:** Thank you for clarifying. I can see the improvement of the AUROC now. **W6:** Thank you for the additional experiments. Even though these experiments are done only with a very small sample size, I can see the data replication. **Q3&Q4:** Thank you very much for the additional experiments. The results with different triggers seem to be consistent with the results in the paper. **Q5 & Q6:** Thank you for the additional insights. The rebuttal has answered all my questions. Based on the rebuttal, I will raise my score accordingly.