Direct Unlearning Optimization for Robust and Safe Text-to-Image Models

Summary

The paper proposes a new method for unlearning diffusion-based generative models. The proposed method uses the basic idea of reinforcement learning with human feedback. To collect pairs of images to be unlearned and corrected for preference optimization, the author uses SDEdit to modify the NSFW content. Experimental results show that the proposed model successfully unlearns harmful content from the generated images.

Strengths

The use of SDEdit for curating semantically similar paired images sounds novel and effective.

Weaknesses

- Although this paper shows some promising results about diffusion model unlearning, the main technical contribution is limited to a combination of SDEdit and DiffDPO. - It is clearly an overstatement that the preference optimization does not affect the quality of the unrelated concepts (in line 149). - I have concerns about the reproducibility of this work. - Minor - The main text and appendix have multiple typos and require careful proofreading. - Typo: Eq 17 → Eq 11 in the main text - Type: line 275: lambda? - Is DCO in line 285 a typo?

Questions

- The reason why the gradient ascent term reduces the model's denoising ability, even in the presence of the KL term, is unclear. Could you provide more details on how adding a L_prior loss, which minimizes the difference between denoising directions at time T, can address this issue? The explanation is insufficient to grasp the logic behind the loss. - What does ‘prior’ mean in ‘prior preservation performance’? If LPIPS measures the perceptual similarity between images, can we just use perceptual similarity? Since the prior in ‘prior preservation performance’ and the prior in ‘L_prior’ are different, it’s a bit confusing to follow section 4.3.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are well-addressed.

Summary

This paper proposes a diffusion unlearning optimization framework to achieve NSFW visual content removal in T2I models. Specifically, the authors develop an image-based unlearning method that utilizes curated paired image data (unsafe images and their corresponding safe images generated by the SDEdit model) for preference optimization. Additionally, they introduce a regularization term to preserve the denoising capability of the diffusion model. This work claims to be the first to apply preference optimization to the unlearning problem. Experimental results demonstrate that their method effectively removes unsafe visual concepts without significant performance degradation on unrelated topics.

Strengths

1. The model is well-explained, and the paper is well-written. 2. The novel perspective of applying preference optimization to the unlearning problem is interesting and promising. 3. The red teaming results are encouraging.

Weaknesses

1. The authors only fine-tuned Stable Diffusion v1.4 for evaluation, which employs U-Net for denoising. However, other popular text-to-image (T2I) diffusion models, such as PixArt[1] and Stable Diffusion v3, which use Transformers for denoising, are not mentioned. It is somewhat limited in discussing the proposed method on only one T2I model. Can the proposed method be applied to the above models? 2. The unsafe concept may introduce some unrelated visual features when generating images, such as image details. How can the authors ensure that the proposed method does not affect these unrelated features when removing unsafe content? [1] PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR 2024

Questions

Please refer to the above weakness section.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

Refer to previous sections.

Summary

The authors introduce synthesized image data and preference optimization for concept unlearning in diffusion models. Additionally, they consider the regularization of model preservation performance to ensure a balanced approach.

Strengths

1. The presentation is clear and well-structured. 2. There is a solid theoretical basis for the proposed preference optimization.

Weaknesses

1. The efficacy of the proposed method is heavily dependent on the quality and diversity of the synthesized image pairs. The utilization of a small dataset, consisting of only 64 pairs, raises significant concerns about the potential for overfitting. To enhance the credibility of the findings, it is imperative for the authors to conduct comprehensive ablation studies. These studies should explore the impact of different sets of synthesized image pairs on model performance, potentially revealing critical insights into the method's robustness and generalizability. 2. The current robustness assessments of the study are limited by the employment of relatively weak attack scenarios. For a more robust evaluation of the unlearned models, it is crucial to incorporate a stronger, white-box attack methodology. The use of UnlearnDiffAtk [1], a commonly applied tool for assessing the robustness of unlearned diffusion models, is recommended. This approach would not only adhere to contemporary research standards but also significantly bolster the validity of the results. Furthermore, detailed reporting of the attack success rate (ASR) associated with UnlearnDiffAtk would provide a more precise quantification of the models' resilience against sophisticated attacks. [1] "To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy to Generate Unsafe Images ... For Now", ECCV 2024.

Questions

Check the weaknesses section for more details.

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Effectiveness of the proposed method heavily relies on the synthesized image pairs and different synthesized image pairs might cause high performance variance.

Summary

The authors address the issue of adversarial attack to diffusion model generating inappropriate image contents in this paper, critiquing the previous work on unlearning technique unlearns harmful prompt but making themself vulnerable to adversarial prompt. Instead they propose the image-based unlearning technique to remove unsafe visual features while retain the performance to generate image for safe concepts. During the training, they propose the method to generate a pair of safe and unsafe content and employ preference optimization to encourage diffusion model to generate the safe counterpart. The present the result in the evaluation to show that their model is better at defending adversarial attacks such as SneakyPrompt, Ring-A-Bell and Concept Inversion.

Strengths

- The paper is is structured in a clear and logical manner, making it easy to follow. Both the introduction and the related work sections are well-written providing comprehensive context for the problem and the clear overview of previous work. - The necessity of addressing this problem is convincingly justified, and the proposed solution is closely aligned with this motivation. The derivation of the proposed method is elaborated in detail and is technically sound. - The authors demonstrate strong results in the evaluation and show convincing results successfully defend adversarial attack. The present ablation study to prove the effectiveness of output preserving regularization.

Weaknesses

- There is a missing detail in experiments setup. The authors mention in line 194 that they decompose the violence concept into four categories and apply DUO to each of them and then merge the final models. If I understand correctly, the authors apply DUO independently for the four concepts resulting in four final models with LoRA. Please clarify how merging works to combine the four models.

Questions

Please clarify the missing detail

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discuss the limitation in the conclusion.

Reply to Authors

Thanks for your reply. The authors have addressed my concerns. After considering the comments from other reviewers and your reply, I plan to keep my rating.

Read the rebuttal and discuss with the authors

Hi Reviewers, The authors have submitted their rebuttal responses addressing the concerns and feedback you provided. We kindly request that you review their responses and assess whether the issues you raised have been satisfactorily addressed. If there are any areas where you believe additional clarification is needed, please do not hesitate to engage in further discussion with the authors. Your insights are invaluable to ensuring a thorough and constructive review process. Best AC