PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher

Summary

The paper proposes PaGoDA, an adversarial distillation method to support single-step generation of image resolutions higher than the teacher diffusion model. PaGoDA first solves the forward PF-ODE (data to noise) to collect noise from the dataset. Then, it gradually adds upsampling layers to the model and trains only those layers. The training objectives include reconstruction loss and GAN loss, and for text-to-image experiments, additional distillation loss and CLIP regularization are introduced to support classifier-free guidance. PaGoDA demonstrates strong one-step generation performance on ImageNet and COCO benchmarks and shows its superiority through various ablation studies.

Strengths

**Originality** - This is the first study to support higher resolution than the teacher diffusion model when distilling a diffusion model into a single-step generator. - It is also the first study to use the forward PF-ODE to collect noise-data pairs for distillation. **Significance** - Despite using GAN loss, the training is stable due to the progressive growing strategy, as shown by the good performance at various resolutions in Table 2. - By using the forward PF-ODE for distillation, it learns the true data distribution and achieves higher sample diversity compared to GANs, as evidenced by the recall performance in Table 2. - The PaGoDA is shown to be applicable to the pixel-space text-to-image model, DeepFloyd. **Quality** - The paper is well-written and includes detailed implementation details in the appendix for reproduction. - Various ablation studies demonstrate the superiority of each loss and the upsampling method of this approach compared to the VAE decoder of latent diffusion models.

Weaknesses

- PaGoDa is claimed to have a faster sampling speed compared to LCM, but wouldn't the sampling speed be faster with **smaller VAE** [A]? Wouldn't LCM be more efficient than PaGoDa if the VAE decoder has less layers than PaGoDa's additional upsampling layers? - Additionally, smaller VAE only requires training of a VAE, while PaGoDa involves sequential training stages depending on the resolution, making the training pipeline somewhat inconvenient. - In Table 4, PaGoDa performs better than the diffusion teacher, and in Table 1, PaGoDa outperforms EDM2-XXL. This may be because **the discriminator uses EfficientNet and DeiT pre-trained on ImageNet**, leading to better FID scores, which use an Inception network also trained on ImageNet. This issue has been discussed in [B]. There are two potential solutions to this problem. The first is to compare **Fréchet distances using DINOv2**, following the EDM2 paper. The second is to **train the discriminator for GAN loss from scratch**. If PaGoDa's GAN loss is truly stable, it should be possible to train without a pre-trained network. [A] Tiny AutoEncoder for Stable Diffusion: https://github.com/madebyollin/taesd [B] Kynkäänniemi et al., The Role of ImageNet Classes in Fréchet Inception Distance, ICLR 2023.

Questions

- What classifier-free guidance scale did PaGoDA use in Table 6? Is the point that achieved the best performance in Figure 14 reported in Table 6? - Why were different datasets used when collecting latents (CC12M) and during GAN training (COYO-700M)? - Does the DDIM inversion in Section 5.4 needs a diffusion teacher for inversion?

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Appendix C explains well the gap between PaGoDA's theoretical assumptions and its practice.

Summary

The paper introduces a novel GAN-based diffusion distillation approach that leverages ideas from PG GAN for effective high-resolution synthesis.

Strengths

* PaGoDA demonstrates state-of-the-art single-step generation performance on ImageNet, competing with or outperforming more expensive alternatives, including the teacher DMs. * The proposed classifier-free GAN is an interesting modification for better conditional generation. * The authors justify most components of the proposed method by providing valuable discussions and ablation studies. * The paper includes thorough theoretical analysis, as well as many experimental and implementation details.

Weaknesses

* PaGoDa is designed for pixel-based DMs only and hence is not applicable to most state-of-the-art T2I models, e.g., SDXL, SD3. Also, the motivation behind this (L42) seems a bit arguable. For example, one could try to adapt PaGoDa to LDM by replacing the AE decoder with a shallow one mapping a 64x64 latent variable to a 64x64 image. Then, PaGoDA can be applied as is. Do the authors have any thoughts on this? * The method requires collecting the dataset of a large amount of (x, z) and ($\hat{x}_w$, z) pairs. Likely, this significantly increases the training costs compared to sampling-free alternatives. Could the authors provide the data collection vs training costs? How does the overall training time compare to other distillation approaches? * The KD objective is used for the guided generators (i.e., using the reverse process for sampling ($\hat{x}_w$, z) pairs). This contradicts the motivation behind the reconstruction loss (L89). While the ablation study in Tab.7 aims to address this, could the authors clarify the inference CFG scale used in this experiment? How does the role of the reconstruction loss change with different CFG scales? * PaGoDa uses CLIP regularization in the T2I setup. I believe this makes the CLIP score evaluation incorrect. Moreover, other distillation methods, e.g., LCM, DMD, or ADD, do not use it. This raises concerns about the performance of the core approach in the T2I setting. * For T2I evaluation, automated metrics have been shown to correlate poorly with human preference (e.g., [1,2]). It would be highly beneficial to conduct a human study comparing PaGoDA to other distillation approaches and T2I diffusion models. If a reliable human evaluation setup is unavailable, I would suggest evaluating a metric learned to mimic human preference, e.g., [3,4,5]. * The CFG details for T2I inference are missing. What scale is in the final t2i results? How does the model perform with different CFG scales? * The paper does not provide qualitative T2I results demonstrating the diversity of generated images for a given prompt. * The ablation study lacks evaluation of the proposed approach without adversarial objective. [1] Podell et al. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. [2] Li et al. Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation. [3] Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation [4] Kirstain et al. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. [5] Wu et al. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. [6] Sauer et al. Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation

Questions

* Please address the questions and concerns raised in "Weaknesses". * Intuitively, the reconstruction and distillation losses are most beneficial for the base-resolution training. Did the authors try to deactivate these losses for SR stages? * Did the authors consider using a pretrained DM as a discriminator, following the ideas from [6]? It would be interesting to explore if noise-level feedback might be valuable for learning better generators. * PaGoDa approaches very low FID values on ImageNet. Could the performance gains be a result of overfitting the training data?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed the limitations and potential negative social impact in the appendix.

Summary

This paper introduces a method to distill a diffusion model into a one-step generator. The training process combines several loss functions. First, the authors use DDIM inversion to transform real images into latent noise, which is then fed into the generator. The generated images are supervised with a reconstruction loss. Second, an additional GAN loss is applied to the generated images, distinguishing them from real images. For text-to-image generation, the authors present a novel approach to enable classifier-free guidance in the GAN loss by using an auxiliary guidance predictor. Other auxiliary losses, such as knowledge distillation loss and CLIP loss, are also employed to enhance performance. The final method are evaluated across various benchmarks.

Strengths

S1. The presented approach performs well on common benchmarks while achieving good inference efficiency. S2. The authors plan to open-source the method, which will be beneficial for further research.

Weaknesses

W1. Some loss components are questionable and may mislead readers about the actual performance of the proposed approach. For instance, it is well-known that using a GAN loss with a discriminator pretrained on ImageNet significantly biases the FID metric [1]. This issue also applies to the CLIP loss, which biases the evaluation of FID/CLIP scores when used during training. A substantial portion of the strong results presented might be attributed to these misleading numbers, making it difficult to draw accurate conclusions about effectiveness. This is a common issue in some previous works but needs to be addressed now. For example, using GANs is good, but it might be better to avoid using a pretrained classifier as the discriminator, as suggested by recent explorations in diffusion GAN literature [2]. For text-to-image results, it is recommended to conduct human evaluations to assess real image quality, prompt alignment, and diversity. W2. Many of the proposed components have been well explored in previous literature, but the connections to these works are not sufficiently discussed in the related works section. For instance, the combination of a regression loss and a GAN loss has been explored for text-to-image generation in several studies, including [3, 4]. Additionally, the use of distillation loss and CLIP loss are common practices [5, 6]. This reduces the perceived novelty of the paper to a moderate level. W3. The introduction of a classifier-free GAN objective might overcomplicate the problem. Previously, GAN loss with classifier-free guidance (CFG) was enabled by generating fake samples using an original diffusion model with guidance, as applied in [2], and then utilizing the standard GAN loss. This method is simpler than training an auxiliary classifier and has a similar computational cost, since the current classifier training also requires generating samples with CFG using the original diffusion model. [1] Kynkäänniemi, Tuomas, et al. "The role of imagenet classes in fr\'echet inception distance." arXiv preprint arXiv:2203.06026 (2022). [2] Sauer, Axel, et al. "Fast high-resolution image synthesis with latent adversarial diffusion distillation." arXiv preprint arXiv:2403.12015 (2024). [3] Lin, Shanchuan, Anran Wang, and Xiao Yang. "Sdxl-lightning: Progressive adversarial diffusion distillation." arXiv preprint arXiv:2402.13929 (2024). [4] Song, Yuda, Zehao Sun, and Xuanwu Yin. "SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions." arXiv preprint arXiv:2403.16627 (2024). [5] Sauer, Axel, et al. "Adversarial diffusion distillation." arXiv preprint arXiv:2311.17042 (2023). [6] Kang, Minguk, et al. "Scaling up gans for text-to-image synthesis." CVPR 2023

Questions

Most of my concerns are outlined in the weaknesses section. To reiterate, I am particularly concerned about the utilization of a pretrained classifier as a discriminator and the CLIP loss, which bias the quantitative evaluation. Additionally, the overall novelty is moderate, and some components might be unnecessarily complex.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

yes.