Targeted Attack Improves Protection against Unauthorized Diffusion Customization

Response to Reviewer (1/2)

We thank the insightful review and would like to address the issue by points: [1] (W1: lack of recent baselines) We thank reviewers for mentioning new baselines and have included them in our related works in the updated version. We are still doing experiments on SDS and MetaCloak (Influence Watermark has no open-sourced implementation) and have not had their results ready due to the limited time. Also, we find that the mentioned three baselines may not be necessary baselines for our method, where we give the reasons one by one: (SDS) According to its abstract [A], SDS mainly focuses on doubling the speed of protection and reducing memory occupation by half without compromising its strength. This indicates that the focus of SDS is mainly on computational efficiency. While our focus is to improve the protection effectiveness and ACE shows clear superiority compared to AdvDM, which is the main baseline of SDS, we believe ACE could outperform SDS. (MetaCloak) MetaCloak mainly focuses on robustness to Gaussian transformation. According to Figure 2 in [B], its protection effectiveness is inferior to ASPL under no transformation scenarios. On the other hand, ACE outperforms ASPL clearly according to our experiments. Hence, we believe that ACE could outperform MetaCloak under no transformation scenarios. Also, it is possible that we can embed ACE in the objective term of MetaCloak to yield both good effectiveness and robustness. (Influence Watermark) Influence Watermark focuses on protection effectiveness. However, it has neither open-sourced implementation nor information for reproduction (even the adversarial budget is missing). Hence, we can only provide some insights which may help the reviewer to compare our method with it. Two metrics are shared on the same dataset for Influence Watermark and ACE: FDFR and ISM, which mean the failure ratio of generated portraits being detected as faces and the ratio of generated portraits being detected as the correct identity. Our FDFR is 1.0, that none generated portraits are detected as faces and ISM is NA, because none generated portraits are detected as faces with some identity. **This is the upper bound of protection effectiveness measured by these two metrics**. By contrast, the FDFR and ISM of Influence Watermark are 0.28-0.67 and 0.31-0.33, respectively [C]. While Influence Watermark does not provide the adversarial budget, ACE uses the lowest budget of all existing works, 4/255. Hence, we believe that ACE has sound advantages in effectiveness compared to Influence Watermark. Generally speaking, SDS and MetaCloak do not focus on improving the protection effectiveness, while Influence Watermark seems to be inferior to our method in protection effectiveness. Although these works contribute to the field from different perspectives, we believe ACE has its unique significance in effectiveness. We have updated this discussion on baseline selection in our Appendix A (L788-L797) to make it acknowledged. [2] (W2: experiments on DiT) We conducted new experiments on Stable Diffusion 3, which is the state-of-the-art DiT-based diffusion model. The details are given in Appendix B.4 (L1026-L043). We find that ACE can still degrade diffusion customization on SD3, but in a different way: Instead of degrading the image quality, ACE makes images irrelevant to the base training dataset. As shown in Figure 7 in our new draft, images generated by LoRA under ACE cannot generate portraits with the same person in the training dataset. ACE is the first attack that proves to work on Stable Diffusion 3. [3] (W3: specifical purification) First, our robustness experiments cover more purification methods than the union of our baselines e.g. DiffPure. We believe this is enough to show the robustness of our methods. Second, specifical purification is a novel challenge to the robustness of all protections. To overcome this requires also specifical efforts in designing the adversarial constraints and patterns, which could be a different topic to study. Notably, Good Protection=Effective Objective + Robust Pattern. While our focus is mainly on the objective, we would like to leave countering specifical purification to future work. Finally, we understand the reviewer’s concern in the potential overclaim of our real-world effectiveness. To clarify this, we add the following statement in our limitation (Appendix D, L1200-L1209)

Response to Reviewer

We thank the insightful review and would like to address the issue by points: (1) (lack of novelty): First we would like to point out that PGD attack has nearly become a standard approach for many modern adversarial attacks. Many works in this field adopt PGD as their base attack method [1][2][3][4]. In the field of adversarial attack against diffusion models, our method is fundamentally different from previous works. Though the idea of setting a target for attack is already proposed by [1], the performance of targeted attack still lags behind untargeted ones. We’re the first to advance the performance of targeted attack to SOTA level, even pushing the ASR (Attack Success Rate) on CelebA-HQ to the upper bound: 100\%. Our work also differs from other targeted-attack works from the formulation. Previous works [1][3] tried to pull the flow of diffusion models to a predefined poisoned flow. ACE, in contrast, tried to pull the flow towards a fixed, time-invariant target, which cannot be interpreted as a legit flow. Instead, we formulated this as a consistent score function error over time. This novel view gives targeted attack strong performance, outperforming untargeted ones and reaches the SOTA level. (2) (SD3) We thank the reviewer for bringing SD3 out. As the reviewer mentioned, LoRA on SD3 is not fully supported due to its structure difference to SD1/2. But still as the reviewer requested, we conduct experiments to empirically show that ACE can adapt to SD3 under the LoRA setup. The details are given in Appendix B.4 (L1026-L1043). We find that ACE can still degrade diffusion customization on SD3, but in a different way: Instead of degrading the image quality, ACE makes images irrelevant to the base training dataset. As shown in Figure 7 in our new draft, images generated by LoRA under ACE cannot generate portraits with the same person in the training dataset. ACE is the first attack that proves to work on Stable Diffusion 3. This behavior difference is intriguing, and possibly due to the architecture difference between SD3 and SD1.x. (3) (potential bias) We agree that the Stable Diffusion may already integrate our evaluation data into the training set, as the datasets we use are accessible on the Internet. However, our dataset selection follows the setup of baselines. While some of them have been applied widely, we believe this potential bias does not constitute serious threats to our performance evaluation. Also, Membership Inference Attack (MIA) on large, commercial diffusion models is very hard to conduct, and it’s still an open and challenging problem for academia [5]. Conducting such validation is far beyond the focus of this paper. Finally, we have some positive samples drawn from the artists community, where the attack and evaluation is conducted by the artists on their private artworks. However the artist does not give us the consent to release them. We’ll acknowledge the miss of validation in our next revision. (4) (noisy-upscaling) We have discussed with the authors of ‘Adversarial Perturbations Cannot Reliably Protect Artists from Generative AI’ as soon as it is public. They evaluate ACE on Noisy-Upscaling. Compared to other watermarks like Mist-v1 and Glaze, ACE still survives Noisy-Upscaling to some extent that the produced images include some chaotic contents, for example, mixed candle bodies and flame. Considering that most of the watermarks have been broken by noisy-upscaling, ACE is the only one that keeps some effectiveness. However, due to the double-blind rule, we cannot share the image example here because it could be used to detect our paper. In addition, while our focus is mainly on an effective objective, we just align our robustness experiments to our baselines. We agree that there are specifical purification methods that raise true threats to protection methods like ACE. It is the common challenge of the field, while we believe methods to counter them could be an independent topic to study. Therefore, we would like to leave it to future work. Questions: [1] Please see Weakness (3). [2] Please see Weakness (4). Again, we thank the reviewer for the insightful review. If you have further questions, feel free to contact us. [1] : Van Le, Thanh, et al. "Anti-dreambooth: Protecting users from personalized text-to-image synthesis." [2]: Ahn, Namhyuk, et al. "Imperceptible Protection against Style Imitation from Diffusion Models." [3]: Liang, Chumeng, et al. "Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples." [4]: Chen, Yipu, Haotian Xue, and Yongxin Chen. "Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policies." [5]: Dubiński, Jan, et al. "Towards more realistic membership inference attacks on large diffusion models."

Kindly request for further discussion

As the rebuttal period draws to a close, we kindly request your engagement in the discussion. We give further discussion about our technical contribution to this field. We also conduct experiments on SD3 to show our effectiveness. Notably, **we’re the first attack to show effectiveness on SD3**. We also discuss the effect of noise-upscaling and the potential bias of our training set. Please don’t hesitate to contact us if you have any further questions. We would sincerely appreciate it if you could consider engaging in further discussions.

Response to Reviewer

We thank the insightful review and would like to address the issue by points: [1] (Difference from Glaze) Unfortunately, Glaze does not have an official open sourced implementation and the demonstration in the paper does not give the details on the components of Stable Diffusion being used and how to use them to apply Glaze. However, based on the information we are acknowledged, we can still conclude two clear technical differences between ACE and Glaze: a) The objectives are different. ACE minimizes the distance between diffusion predicted latent noise and a fixed latent. We have both VAE and UNet involved. By contrast, Glaze minimizes the feature distance between the protected image and the target image in a feature representation space. It seems that the space is the VAE latent space of Stable Diffusion in their implementation. Only VAE is used in this process. b) The target selection is different. ACE exploits the latent of a chaotic-pattern image as the target. Glaze uses the style-transfered version of the original image as the target image. Generally speaking, two of our core contributions, a new objective and the target selection, are both novel compared to Glaze. Although ACE and Glaze are both called targeted attacks, we believe their technical cores are very different. We have added an extra section in Appendix D (L1244-L1264) for more details. In addition, we would like to note that Glaze does not use normal l2-constraint, thus making it impossible to do fair comparisons with all other baselines and our method. Hence, we follow previous works to not include Glaze in our baselines. [2] (performance of partial attacks) This is an interesting topic to investigate, while we believe this is a common issue of all protection methods rather than the specifical issue of ACE. Hence, we believe this is out of the scope of our core contributions on improving the effectiveness of protection methods. Despite this, we conducted a simple but intuitive experiment on SD3 to provide some insights. We merge the clean portrait dataset of person A and a protected portrait dataset of person B. We fine-tuned SD3 LoRA on this merged dataset and found that the fine-tuned LoRA can only generate the portraits of person A. This indicates that LoRA may ignore the protected images and only learn the unprotected images in this scenario. However, due to the time limit, we would like to leave the further exploration to future work. Again, we thank the reviewer for the insightful review. If you have further questions, feel free to contact us.

Response to Reviewer

We thank the insightful review and would like to address the issue by points: [1] (specifical purification) First, our robustness experiments cover more purification methods than the union of our baselines e.g. DiffPure. We believe this is enough to show the robustness of our methods. Second, specifical purification is a novel challenge to the robustness of all protections. To overcome this requires also specifical efforts in designing the adversarial constraints and patterns, which could be a different topic to study. Notably, Good Protection=Effective Objective + Robust Pattern. While our focus is mainly on the objective, we would like to leave countering specifical purification to future work. Finally, we understand the reviewer’s concern in the potential overclaim of our real-world effectiveness. To clarify this, we add the following statement in our limitation (Appendix D, L1200-L1209) In addition, we follow current works to use l2-distance to constrain our adversarial perturbations, which makes comparisons fair. This design, however, **could be vulnerable** to some new advanced purification methods, for example, IMPRESS and GrIDPure. We believe this could be overcome by rectifying the constraint and designing patterns that hide our protection deeper in the original image. However, this is out of the scope of this paper which focuses on optimizing the attack objective. We believe our stronger objective can contribute to real-world protection in combination with a potential robust constraint and pattern, where we leave to the future work. Also, our robustness experiments show that our method has robustness against general purification, keeping its essential utility. Hence, we leave the improvement of the robustness against advanced purification to the future work. [2] (added loss in ACE+) The sole added loss in ACE is the PG in our baselines. We have listed its performance in Table 1, where it does not seem to be competitive compared to other baselines and our methods. Additionally, we add ACE+ mainly to show a variant with comparable performance to ACE and different patterns in the output images, which provides more choices for real-world practice. The added loss mainly serves to change the pattern of ACE rather than helping ACE+’s outperform ACE. Hence, we believe no extra ablation studies are needed. Again, we thank the reviewer for the insightful review. If you have further questions, feel free to contact us.

Kindly request for further discussion

Dear Reviewer, We highly appreciate your constructive review and comprehend your concerns on our robustness to specific purification methods as well as ablation studies. To address these concerns, we updated our paper draft and provided a detailed explanation above. As the rebuttal period draws to a close, we sincerely look forward to your further ideas on these concerns and kindly request your engagement in the discussion. Notably, our newly-conducted experiment shows that **our method is the first attack to show effectiveness on SD3** (See Appendix B.4, L1026-L1043). We also want to hear your advice on the updated content. We would sincerely appreciate it if you could consider engaging in further discussions.