Summary
This paper presents a unified backdoor attack framework (VillanDiffusion) to expand the current scope of backdoor analysis for DMs. The proposed framework covers mainstream unconditional and conditional DMs (denoising-based and score-based) and various training-free samplers for holistic evaluations.
Strengths
1. Their experimental results not only analyzed DDPM but also score-based models. Besides, they also analyzed other acceleration sampling methods.
2. Their experiments included caption triggers.
Weaknesses
1. To show that no modifications are needed to the sampling process, the article should include details of the sampling process.
2. The order of formulas 8-12 in the article is not clear enough. To describe in the order of forward process → backward process → sampling process may be more clear.
3. Please check some spelling errors in the article. For example, "Praobility" in the title on line 138.
4. Please check whether the last term in Eq. 4 is $L_0(x_1,x_0)$.
5. The article claims that BadDiffusion will fail when the coefficient is $\frac{1}{2}$ (line 55), but there is no further explanation.
6. The article claims that an attacker only needs to obtain the model parameters $\theta_{download}$. However, to execute a backdoor attack, some adjustments need to be made to the initial noise. This is difficult to achieve in reality, and the article needs to emphasize this point.
7. Although the article extends the attack to other models, such as score-based models, there is no essential difference from BadDiffusion[2]. On BadDiffusion, just changing the coefficient of the noise term $1-\bar\alpha_t \mathbf{I}$ in the formula $q\left(\mathbf{x}_{t}^{\prime} \mid \mathbf{x}_{0}^{\prime}\right):=\mathcal{N}\left(\mathbf{x}_{t}^{\prime} ; \sqrt{\bar{\alpha}_{t}} \mathbf{x}_{0}^{\prime}+\left(1-\sqrt{\bar{\alpha}_{t}}\right) \mathbf{r},\left(1-\bar{\alpha}_{t}\right) \mathbf{I}\right)$ (Eq. 6 in their article [2]) to $b_k \mathbf{I}$ can easily obtain the results in this article. In addition, Eq. 11 is only a change of sign of Eq. 38 in [1].
[1] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
[2] Chou, S. Y., Chen, P. Y., & Ho, T. Y. (2023). How to backdoor diffusion models?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4015-4024).
Questions
Please see the Weakness part.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors didn’t mention any potential limitations.
The authors can list limitations of the application. For example, to apply their method, the attacker needs to access the initial noise, which is not practical sometimes. Besides, the negative societal impact should also be included.