Summary
The paper proposes a new method to fine-tune the pretrained large-scale diffusion models for new tasks. It finds that different time steps of the denoising process of diffusion models have varying transferability. Specifically, the paper finds that "low-noise" time steps close to the end of the denoising process have better transferability. In contrast, the "high-noise" steps responsible for generating the image layout and containing domain knowledge are less transferable. Based on these observations, the paper proposes a novel fine-tuning objective demonstrating better convergence speed than baselines.
Strengths
- The idea is simple without numerous hyperparameters or tricks to work.
- The proposed idea is orthogonal to the main focus of the recent papers that aim to do parameter-efficient fine-tuning, and it can be combined with them as the paper demonstrates in the experiments.
Weaknesses
1. I could not follow the motivation behind the scheme that the paper claims to perform Knowledge Retention with. The main argument of the paper is that "the time steps of the denoising process close to the end of the sampling chain of diffusion models are more transferable." Then, why do we need to sample an augmented dataset to do the Knowledge Retention? A straightforward way can be using knowledge distillation from the original model to the fine-tuned one in the low noise time-steps.
2. Following the previous point, distillation may be better than an augmented dataset considering that the fine-tuning data may have characteristics that are different from the original domain. For instance, (this is just a hypothetical example) the fine-tuning dataset may be on flowers that are mostly red, but the pretaining dataset (ImageNet in the experiments) may not have red flowers. Therefore, for the fine-tuning process, we care more about red flowers, not other types. However, using an augmented dataset can bias the fine-tuning process and lower the model's quality on red flowers. In contrast, distillation on the low-noise time steps may readily preserve the prior knowledge of the model without biasing the fine-tuning process.
3. I appreciate the analysis in Fig. 1, and the formal experimentation of it is a novelty. Yet, some similar phenomena have been shown in the literature before that the paper does not discuss them. [1] shows that a diffusion model usually generates a high-level layout of the image until SNR gets between [0.01, 1], and then, starts to fill in the details. Similarly, eDiff-I shows that changing the text prompt at the low-noise stages of the denoising process process does not change the output image. These experiments in Fig. 1 also show that if we use the original model in the low-noise time steps, the performance does not decrease, which is similar to these papers. I think the paper should include these references in the related work section.
[1] Perception Prioritized Training of Diffusion Models, CVPR 2022.
[2] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, 2023.
Questions
I have the following questions from the paper:
1. When fine-tuning the ImageNet pretrained model on the new task, do you train the class embeddings (used for cross-attention) for the new task as well?
2. Regarding points 1 and 2 in the Weaknesses section, as far as I know, training on images generated by the generative models may degrade the model's performance [1]. Yet, using the whole ImageNet dataset performs worse than using 5k images that are sampled from the pretrained model. Is there any intuition about why this happens?
[1] Self-consuming generative models go mad, ICLR 2024.
Limitations
The authors have pointed to using an augmented dataset as a potential limitation, but as I mentioned in the weakness section, the motivation and intuition of why it should work is not clear.