Addressing Negative Transfer in Diffusion Models

Summary

This paper discusses the use of diffusion-based generative models for various generative tasks, such as image, video, 3D shape, and text generation. The authors argue that negative transfer, which refers to competition between conflicting tasks leading to decreased performance, should be investigated and addressed in diffusion models. They observe a negative correlation between task affinity and the difference in noise levels, suggesting that adjacent denoising tasks are more harmonious. They also find evidence of negative transfer in diffusion model training, where utilizing a model trained exclusively on denoising tasks within a specific interval generates higher-quality samples compared to a model trained on all tasks simultaneously. To address negative transfer, the authors propose leveraging existing multi-task learning techniques and clustering denoising tasks into intervals. They demonstrate the effectiveness of their approach through experiments on image datasets, showing improved quality in generated images.

Strengths

1, The paper provides a comprehensive analysis of diffusion-based generative models, considering their performance and flexibility in various generative tasks, including image, video, 3D shape, and text generation. It highlights the achievements and potential areas for improvement in diffusion models. 2, The authors identify and address the issue of negative transfer in diffusion models. They observe evidence of negative transfer during model training and propose a clustering approach to mitigate its impact on denoising tasks. This analysis adds valuable insights to the field of multi-task learning and diffusion models.

Weaknesses

1, The experiments conducted in the paper are limited to specific datasets (FFHQ and CelebA-HQ) and specific types of diffusion models (Ablated Diffusion Model and Latent Diffusion Model). This limited scope may restrict the generalizability of the proposed approach to other datasets or types of diffusion models. The paper could have benefited from evaluating the method on a broader range of datasets and models. 2, While the paper presents empirical results to support the effectiveness of the proposed approach, it lacks in-depth theoretical analysis or mathematical formulation of the clustering strategy. A more rigorous theoretical analysis would have provided a deeper understanding of the approach and its underlying principles.

Questions

1, What is the proposed clustering approach for addressing negative transfer in diffusion models? How does it take into account noise levels and task affinity? 2, What are the limitations of the proposed clustering approach in terms of scalability and computational complexity? 3, How did the authors validate the effectiveness of their approach? Can you provide more details about the experimental setup and the results obtained?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

As stated in "Weakness".

Summary

This paper analyzes diffusion training from an MTL perspective and observes the negative transfer in diffusion training. Several multi-task learning algorithms are employed to address the negative transfer problem, leading to improved performance.

Strengths

1. The paper presents a detailed analysis of task affinity and negative transfer across different diffusion steps, which is interesting. 2. The incorporation of MTL methods has yielded a substantial improvement in the performance of the diffusion model.

Weaknesses

1. The paper lacks a comparison of time and GPU memory usage between the vanilla and MTL approaches. Typically, MTL methods require more time and significantly higher GPU memory. 2. It would be beneficial to include random weights and linear scalarization baselines. Several studies have suggested that linear scalarization and random weights can serve as strong baselines [1][2]. [1] Xin et al., Do Current Multi-Task Optimization Methods in Deep Learning Even Help?, NeurIPS 2022 [2] Lin et al., A closer look at loss weighting in multi-task learning, arXiv 2022

Questions

Have you considered utilizing the weights obtained by MTL approaches as the loss weights for training the model? For instance, using the weighted loss with the mean weight of each interval in Figure 3b. Would this approach yield comparable results?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

See weakness.

Summary

This work explores the phenomenon of negative transfer in the diffusion training procedure, where different time steps or signal-to-noise ratios may conflict with each other and degrade overall performance. The authors propose a solution to this problem by introducing internal clustering and implementing several multi-task learning (MTL) methods to mitigate the negative transfer effect. The proposed approach is evaluated through experiments on the FFHQ and CelebA datasets, and the results demonstrate its effectiveness in improving performance compared to existing methods.

Strengths

1. It is interesting and novel to approach the training of diffusion models from the perspective of multi-task learning (MTL). By recognizing the potential for negative transfer in the diffusion training procedure, the authors of this work are able to leverage the benefits of MTL to mitigate the negative effects of conflicting time steps or signal-to-noise ratios. 2. The motivation sounds good and the preliminary experiment can demonstrate the negative transfer phenomenon in diffusion models. 3. After addressing negative transfer through internal clustering and MTL methods, the final model achieve good performance on FFHQ and CelebA. 4. The writing is good and easy to follow.

Weaknesses

My main concern is that the experiments are only conducted on small datasets like FFHQ and CelebA. Now many standard benchmarks exist. e.g., ImageNet-64, ImageNet-256, CoCo. Experiments on more datasets are needed to prove the effectiveness of the proposed methods.

Questions

1. It would be beneficial to include the average conflict of the baseline model in Fig 3(a) to provide a clearer comparison. 2. The choice of the number of clusters is an important parameter in the proposed internal clustering approach, and it would be beneficial to conduct further analysis and ablation studies to determine the optimal number of clusters.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Thanks to reviewer 8Ptq

We deeply appreciate your effort in responding to the rebuttal. We are delighted that your concerns have been resolved.