Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training

Summary

The paper presents a method to improve the parallel sampling of the diffusion model by improving the denoising network for out-of-distribution evaluation. The paper proposes to finetune a trained model using the log-likelihood ratio of a sample at two different noise scales. The log-likelihood ratio is obtained by integrating the denoising error at all noise scales. The proposed method performs better for parallel sampling and performs comparable for sequential sampling on the CIFAR-10, AFHQ, and FFHQ datasets.

Strengths

- The paper is overall well-written and easy to follow. - As the contrastive diffusion loss is expensive to compute, the authors show that finetuning on pretrained model is sufficient for few epochs instead of training using contrastive diffusion loss (CDL) (or linear combination of CDL and denoising error) through out the training.

Weaknesses

- The contrastive diffusion loss (CDL) contains the integration of denoising error over all noise scales. In practice, I assume that the integration will be replaced by a numerical integration therefore, evaluating the proposed CDL loss is expensive. Considering the training/finetuning using CDL loss is expensive, the benefits from the training don’t seem to be significant in the empirical performance. Additionally, very few/no details are provided on how the integration is approximated in experiments/practice. - Some discussions about the related works need to be included. For example, another work on improving parallel [1] is not mentioned/discussed/compared against. Additionally, the derivation of the CDL loss is somewhat similar to some diffusion classifier papers (for example, [2, 3, 4]) therefore they should be cited/discussed. [1] Accelerating Parallel Sampling of Diffusion Models [2] Your Diffusion Model is Secretly a Zero-Shot Classifier [3] Your Diffusion Model is Secretly a Certifiably Robust Classifier [4] Robust Classification via a Single Diffusion Model

Questions

- It is not clear why only 5k samples are used for calculating FID for parallel sampling. The standard practice is to use 50k samples. As the FID depends on the number of samples, it is hard to make sense of the significance of improvement. - Can you experimentally show/quantify the improvement of the denoiser for out-of-distribution evaluation when trained using contrastive diffusion loss (CDL)?

Rating

3

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

yes

Summary

The paper establishes a connection between the diffusion model denoiser and noise classifier through an examination of log-likelihood ratio estimation. The authors introduce a novel loss function, termed Contrastive Diffusion Loss (CDL), designed to encourage diffusion models to explore OOD regions in noise levels. Empirical results indicate that the proposed loss function enhances performance in parallel sampling and demonstrates robustness to different hyperparameters in sequential sampling.

Strengths

* The paper draws an interesting connection between diffusion model denoiser and noise classifier and introduces a novel loss function based upon. * The paper is well-organized and most results are clearly presented.

Weaknesses

* The clarity of the paper could be further enhanced. For instance, in line 86, the authors introduce the notion $\zeta$ to indicate the noise level, while in Appendix A.2, they use $\alpha$ during the derivation. It seems like the authors want to differentiate between noise level and SNR, but the message is not entirely clear here. * The empirical section of the paper is not as strong as the analytical section. In the parallel sampling example, the method shows good performance, but even with the performance improvement, the results are still far from optimal compared to EDM with the default settings. The experiment on deterministic samplers aims to demonstrate that CDL-regularized models maintain a more stable FID score with changes in NFE, but the authors do not explain why CDL helps in this context. The same issue applies to the experiment on stochastic samplers. Overall, I am on the positive side for this paper. I would appreciate it if the authors could provide more intuitive explanations to better connect the analytical and empirical sections of the paper.

Questions

Why did the authors choose fine-tuning with the CDL over pre-training and using CDL as a regularization?

Rating

7

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors addressed the limitations and potential negative societal impact of their work.

Summary

Building on the denoising score matching loss, the paper introduces a novel contrastive learning loss. This loss estimates the log-likelihood ratio between mixture densities with varying noise levels. The contrastive learning loss is then employed to fine-tune the diffusion model, enhancing its ability to estimate out-of-distribution (OOD) data. To validate this approach, the paper demonstrates experimentally that the fine-tuned diffusion model exhibits improved performance in both sequential and parallel sampling, which typically degrade with poor OOD estimates.

Strengths

The derivation of the contrastive diffusion loss is solid and novel. The CDL could potentially inspire more contrastive loss and noise-level data augmentation methods for diffusion model training. Additionally, CDL introduced the asynchronous data pair $(z_\alpha, \beta)$ for training, improving the robustness of the out-of-distribution region, especially during sampling. Overall, the idea is very sound and novel.

Weaknesses

The experimental results are not particularly impressive: 1. Although CDL fine-tuning consistently enhances parallel sampling performance on both synthetic and real-world datasets, the improvements are marginal. Additionally, the baseline chosen for comparison is not state-of-the-art. For instance, in the unconditional generation in Table 2, even with CDL fine-tuning, an FID of around 7.0 on CIFAR-10 falls short of the state-of-the-art FID, which is less than 2.0. The improvement in sequential generation is even less substantial. 2. I believe the proposed CDL has significant potential. For instance, when addressing inverse problems, the diffusion model also encounters OOD issues. The paper could benefit from including more promising experiments to demonstrate how CDL enhances the diffusion model's OOD estimation capabilities.

Questions

1. I think there are issues with the chosen baseline in the paper. In the original parallel sampling method [27], parallel sampling achieves similar performance to sequential sampling in significantly less time. However, the results reported in Table 2 show that unconditional parallel sampling performs much worse than sequential sampling (around 3.0 for the original DDPM, VP, and VE). Could the author provide more clarification on this discrepancy? 2. According to [1], the denoising score matching loss serves as an upper bound for the negative log-likelihood. So minimizing the denoising score matching loss is equivalent to maximizing the negative log-likelihood. Are there any insights or proofs provided on why minimizing the CDL would also lead to meaningful generation? [1] Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. "Maximum likelihood training of score-based diffusion models." Advances in neural information processing systems 34 (2021): 1415-1428.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have covered the limitations of their work.