Transfer Learning for Diffusion Models

Diffusion models, a specific type of generative model, have achieved unprecedented performance in recent years and consistently produce high-quality synthetic samples. A critical prerequisite for their notable success lies in the presence of a substantial number of training samples, which can be impractical in real-world applications due to high collection costs or associated risks. Consequently, various finetuning and regularization approaches have been proposed to transfer knowledge from existing pre-trained models to specific target domains with limited data. This paper introduces the Transfer Guided Diffusion Process (TGDP), a novel approach distinct from conventional finetuning and regularization methods. We prove that the optimal diffusion model for the target domain integrates pre-trained diffusion models on the source domain with additional guidance from a domain classifier. We further extend TGDP to a conditional version for modeling the joint distribution of data and its corresponding labels, together with two additional regularization terms to enhance the model performance. We validate the effectiveness of TGDP on both simulated and real-world datasets.

Paper

References (54)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer brAM6/10 · confidence 4/52024-06-25

Summary

This paper introduces a new framework, the Transfer Guided Diffusion Process (TGDP), for transferring a pre-trained diffusion model from the source domain to the target domain. They connect the score function for the target domain and the score function of the source domain with a guidance term related to the density ratio. They use a classifier to estimate the density ratio and extend TGDP to a conditional version. Besides, Cycle Regularization term and Consistency Regularization term are proposed to improve the performance.

Strengths

1. This paper introduce a novel training paradigm for transferring a pre-trained diffusion model from the source domain to the target domain in a more efficient and effective way. The proposed guidance term proposed in this paper is a technique worthy of reference and further in-depth study by researchers. 2. The paper is well-written and easy to understand. The theoretical analysis of the robustness is interesting and well described, which helps to understand the proposed techniques. 3. Experimental results show the effectiveness of the proposed method.

Weaknesses

1. The part of Extension to the Conditional Version could be more detailed. The authors could discuss what is the Lemma 3.2 in the conditional version. 2. This paper is missing a comparison with articles related to transfer learning for diffusion models. In experiment part, the authors only compare the TGDP with vanilla diffusion model and finetune generator. 3. Except Gaussian mixture simulations and benchmark electrocardiogram (ECG) data, the authors could provide more experiment results on other datasets.

Questions

1. Is there a writing mistake in Line.162? It is hard to sample from p(x0|xt) rather than q(x0|xt)? 2. What is the Lamma 3.2 in the conditional version? 3. Do you find out other articles related to transfer learning for diffusion models and can you compare other methods with TGDP?

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

A limitation of this study is the lack of empirical validation regarding TGDP’s performance on language vision tasks.

Reviewer UUwS5/10 · confidence 4/52024-07-07

Summary

This paper addresses the transfer learning problem for diffusion models, specifically adapting pre-trained diffusion models to downstream datasets, particularly when the data size is small. Traditional parameter-efficient fine-tuning methods often use pre-trained models as parameter initialization, selectively updating parameters based on prior knowledge. These methods, however, can be suboptimal and not robust across different downstream datasets. The authors propose a novel approach called Transfer Guided Diffusion Process (TGDP), which aims to transfer pre-trained diffusion models more effectively. Instead of fine-tuning, TGDP treats the knowledge transfer process as guided by the pre-trained diffusion model. This method involves learning a domain classifier and using its gradient to guide the estimated score function from the source to the target domain, complemented by additional regularizations in practical implementation. Experimental results demonstrate that TGDP outperforms traditional fine-tuning methods, achieving state-of-the-art performance on both synthetic and real-world datasets.

Strengths

1. Provides a novel perspective on knowledge transfer for pre-trained diffusion models. 2. Theoretical foundation of the proposed method is well-constructed. 3. The paper is well-organized with clear presentation.

Weaknesses

1. **Scope of Title:** - The title "Transfer Learning" is too broad. The paper focuses solely on (few-shot and supervised) domain adaptation, where the upstream and downstream tasks are similar in nature. General transfer learning encompasses a wider range of tasks, including those with different downstream tasks from the upstream one. For instance, transferring pre-trained text-to-image models to controllable generation or text-to-video generation are broader and more complex tasks that fall under transfer learning. - The paper does not address transfer learning across different label spaces. Even within the same generation tasks, domain adaptation requires identical label spaces for source and target domains, which is not always practical. For example, transferring pre-trained conditional generation models on ImageNet to other fine-grained datasets with different label spaces presents a more challenging problem than the domain adaptation addressed in this paper. 2. **Empirical Results:** - The benchmarks used are insufficient. The authors conduct experiments primarily on synthetic datasets and a single real-world dataset (ETG). This limited scope is inadequate for demonstrating the method's effectiveness. More experiments on various modalities and datasets, such as the DomainNet dataset, which is a benchmark for domain adaptation, are necessary to showcase the generalization ability of the proposed method. - The provided analyses are insufficient. While several ablation studies on simulations are included in Appendix C.2, they are not comprehensive. Essential missing analyses include (1) examining the main term $\mathcal{L}\_{\text{guidance}}$ alone and (2) combining $\mathcal{L}\_{\text{guidance}}$ with $\mathcal{L}\_{\text{cycle}}$. These studies are crucial for understanding each term's effectiveness. Additionally, ablation studies should also be conducted on real-world datasets, not just synthetic ones.

Questions

1. **Consistency Regularization:** - The consistency regularization requires optimization of the gradient term $\nabla_{x_t}\log h_\psi(x_t,t)$, but the paper lacks details on how to optimize this term, especially when second-order gradients are involved. Could the authors elaborate on the optimization process for this term? 2. **Effectiveness of TGDP:** - In Table 1, although TGDP significantly outperforms baselines, its performance is highly dependent on the target data size $n$. This seems contradictory to two points mentioned by the authors: (a) the training of the domain classifier $c$ is not significantly affected by $n$ (Figure 2), and (b) the training of the guidance network $h$ with the main term $\mathcal{L}_{\text{guidance}}$ does not require target data (Equation 8). How do the authors explain this discrepancy? 3. **Explanation of Figure 3:** - Figure 3 shows that the visualizations of *Finetune Generator* and *TGDP* appear similar, yet their performance differs significantly. Could the authors provide more insights and explanations about the figure and the reasons behind this performance difference?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes.

Reviewer UUwS2024-08-12

Thank you for your response. However, some of my concerns still remainsunsolved. - **Q1: The title "Transfer Learning" is too broad.** One typical advantage of fine-tuning and other transfer learning methods (compared with score shaping) is to adapt to a new downstream task. For example, stable-diffusion is a *text-to-image generation* model, and [1-2] transfer it to *controllable generation* and *text-to-video generation*, respectively. Could the authors provide further details about how TGDP handles these tasks? - **Q7: Explanation of Figure 3.** Why does using the same encoder learn on target samples lead to similar T-SNE visualizations, although the diversity and fidelity are improved? Are there any results following "change the encoder to the learned classifier"? [1] Adding Conditional Control to Text-to-Image Diffusion Models, ICCV 2023 \ [2] Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models, CVPR 2023

Authorsrebuttal2024-08-12

We thank the reviewer for the comments and suggestions. We appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided. Q1: The title "Transfer Learning" is too broad. One typical advantage of fine-tuning and other transfer learning methods (compared with score shaping) is to adapt to a new downstream task. For example, stable-diffusion is a text-to-image generation model, and [1-2] transfer it to controllable generation and text-to-video generation, respectively. Could the authors provide further details about how TGDP handles these tasks? Thank you very much for your question. Given a source distribution $(x, \boldsymbol{c}_t) \sim p$ ($\boldsymbol{c}_t$ denotes text prompts by using the terminology from [1]), a pre-trained diffusion model can be trained on the source distribution. Given a target distribution $(x, \boldsymbol{c} _t, \boldsymbol{c} _{\mathrm{f}}) \sim q $ ($\boldsymbol{c} _t $ denotes a task-specific condition), [1,2] can be fine-tuned by noise matching objective, $$ \mathcal{L}= \mathbb{E} _{\boldsymbol{x}_0, \boldsymbol{t}, \boldsymbol{c}_t, \boldsymbol{c} _{\mathrm{f}}, \epsilon \sim \mathcal{N}(0,1)}\left[\| \epsilon-\epsilon _\theta\left(\boldsymbol{x}_t, \boldsymbol{t}, \boldsymbol{c}_t, \boldsymbol{c} _{\mathrm{f}}\right) \|_2^2\right]. $$ In our work, we propose directly estimate $\nabla _{\mathbf{x}_t} \log \mathbb{E} _{p(\mathbf{x}_0|\mathbf{x}_t, \boldsymbol{c}_t)}\left[\frac{q(\mathbf{x}_0, \boldsymbol{c}_t, \boldsymbol{c} _{\mathrm{f}})}{p(\mathbf{x}_0, \boldsymbol{c}_t)}\right]$ rather than fine-tuning by noise matching objective. We can still use domain classifier $c_w(x, y)$ to estimate the density ratio, where $y$ denotes the embedding of the condition. Moreover, we would like to discuss some similarities and differences between our work and [1,2]. Directly fine-tuning the diffusion model on data from the target domain is similar to the consistency regularization proposed in our work, while [1,2] has a more in-depth design for the architecture and has great results on vision-language tasks. However, with limited data from the target distribution, direct fine-tuning may not achieve good enough performance, which we verify in a two-dimensional Gaussian setting. Q7: Explanation of Figure 3. Why does using the same encoder learn on target samples lead to similar T-SNE visualizations, although the diversity and fidelity are improved? Are there any results following "change the encoder to the learned classifier"? Thank you very much for your question. When computing FID/Diversity, we use the same encoder to get the feature map, which is the common practice for fair comparison. Therefore, we use the same feature map for T-SNE visualizations, which are generated by the same encoder. For the downstream task, we train a separate encoder for classification.

Reviewer UUwS2024-08-13

Thanks for your response. **For Q1**, I appreciate the potential of TGDP to adapt to more complex tasks and the discussion on their similarities and differences. But I'm still confused about technique details. Could the authors provide more details about the design of $c_w(x,y)$ in case when $c_f\notin\\\{c_t\\\}$? In this case, should $y$ stands for the embedding of $c_f$ or $\\{c_f,c_t\\}$? Since [1] is also applicible to the case with limited data, I expect the authors could provide a comprehensive solution or discuss it in the limitation section. **For Q7**, I understand which encoder to use, but still concerned ***why Figure 3(b) and Figure 3(c) looks similar*** (even if there is an improvement over both diversity and FID). Could the authors provide more details? [1] Adding Conditional Control to Text-to-Image Diffusion Models, ICCV 2023

Authorsrebuttal2024-08-13

We thank the reviewer for the comments and suggestions. We appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided. **Q**: *For Q1, I appreciate the potential of TGDP to adapt to more complex tasks and the discussion on their similarities and differences. But I'm still confused about technique details. Could the authors provide more details about the design of $c_w(x,y)$ in case when $c_f\notin\{c_t\}$? In this case, should $y$ stands for the embedding of $c_f$ or $\{c_f,c_t\}$? Since [1] is also applicible to the case with limited data, I expect the authors could provide a comprehensive solution or discuss it in the limitation section.* **A**: Thank you very much for your question. In [1], text prompts $c_f$ and a task-specific condition $c_t$ can have different modalities, e.g., image condition. We believe TGDP is still general enough to deal with this case. We propse a possible way in the following: 1) When $c_t$ has the same data modality as $c_f$, we can use the same pre-trained Language Model to generate the embedding, i.e., $y_{\text{source}}=\text{Encoder}(c_f)$ for the data from the source domain and $y_{\text{target}}=\text{Encoder}(c_f)+\text{Encoder}(c_t)$ for the data from the target domain. 2) When $c_t$ has different data modalities as $c_f$, e.g., image condition, we rely on the aligned encoder to generate the embedding, e.g., ALIGN [Jia et. al. 2021] in vision-language tasks, i.e., $y_{\text{source}}=\text{Encoder}_1(c_f)$ for the data from the source domain and $y _{\text{target}}=\text{Encoder} _1(c_f)+\text{Encoder} _2(c_t)$ for the data from the target domain. Then, the domain classifier $c_w(x,y)$ is used to distinguish the $(x_{\text{source}}, y_{\text{source}})$ and $(x_{\text{target}}, y_{\text{target}})$, thereby estimating the density ratio. The approach in [1] addresses limited data scenarios by using zero convolution layers (i.e., 1×1 convolution layers with both weight and bias initialized to zero), which mitigates instability during fine-tuning. This is distinct from our methodology, which leverages the smaller sample complexity of the classifier/density ratio estimator compared with the diffusion model. We will include a discussion of the similarities and differences with [1] in the revised version. **Q**: *For Q7, I understand which encoder to use, but still concerned why Figure 3(b) and Figure 3(c) looks similar (even if there is an improvement over both diversity and FID). Could the authors provide more details?* **A**: Thank you very much for your question. We observe a significant difference in the distribution patterns of the purple and black classes. In Figure 3(c), the black class is more concentrated compared to Figure 2(c), suggesting that the synthetic data may lie on the correct manifold. However, given that T-SNE reduces a 128-dimensional feature space to 2 dimensions, we must be careful not to overinterpret these visualizations. The numerical results will provide a more reliable basis for conclusions. [Jia et. al. 2021]: Jia, C., Yang, Y., Xia, Y., Chen, Y., Parekh, Z., Pham, H., Le, Q.V., Sung, Y., Li, Z., \& Duerig, T. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. ICML 2021.

Area Chair uJk72024-08-12

reminder to authors to reply before the end of the author/reviewer discussion period

Hello authors of the paper, It would be great if you could respond to reviewer UUwS's most recent comments before the author/reviewer discussion period closes (at Aug 13, 11:59pm AoE). Thanks, Your AC

Authorsrebuttal2024-08-12

We thank AC for the kind reminder. We appreciate the time you spent on the paper.

Reviewer UUwS2024-08-13

Thanks for your response and my concerns are addressed. I appreciate the contribution of TGDP in motivation novelty and theoretical soundness, but yet expect the scope to be more clear (since the title is *Transfer Learning*), and believe more discussions about TGDP on other fine-tuning scenarios could make the paper better. Above all, I raise my score to 5, i.e., a borderline score.

Reviewer 53Zj6/10 · confidence 3/52024-07-11

Summary

The paper proposes an approach for transfer learning based on score-based generative models and density-ratio estimation. The authors show that in order to transfer a trained diffusion model from a source to a target domain, all that is needed is the expectation of density ratios of source and target domains. This expectation can be learned using neural networks reducing the problem of learning a generative model on the target domain to a regression/classification task.

Strengths

- As far as I can judge, the proposed method for transfer learning is both original and significant. It should make a good contribution to the research community. - In addition to the core of the method, the proposed regularizers that improve efficiency are in my opinion conceptually convincing. - The paper is well written and easy to follow. - The experimental evaluation on the ECG data is in my opinion well executed and convincing.

Weaknesses

- While the method is in my opinion theoretically convincing, the experimental section and ablations are a bit lackluster which decreases confidence in the method (even though I acknowledge the complexity of the task itself). - The first experiment on a 2D Gaussian illustrates the performance of the method in comparison to baselines decently, but it is in my opinion ultimately too trivial to be really meaningful (since the density ratios of the 2 Gaussian mixtures can be easily learned, because they are well separated, I would expect that a method based on learning density ratios outperforms baselines here). The ablation in Figure~4 in the appendix is conducted too superficially. It consists of scatterplots, no real quantitative analysis, and doesn't evaluate all relevant loss terms of the model. - The evaluation on the ECG data is not fully clear, since meta-information, such as sampling rate, and other details are not given (as far as I can tell). - In my opinion, a quantitative evaluation on at least 2-3 more high-dimensional real-world data sets should have been conducted in addition.

Questions

- Figure 1 shows the data as scatterplot which makes it impossible to see the learned density. Since, the true densities are known, a representation using kernel density estimates would be better to see if the learned model aligns with the truth. - For readers who do not know what a "12-lead ECG data" is, the supplement should contain this information (at least a rough overview). How does 12-lead ECG data look? Are they one-dimensional time series? At what rate are they sampled? - Task 4.2.2 is a bit unclear. The authors should be more detailed what the goal is and what they are exactly doing (even though one can infer it from the text). - It would have been nice to see the ablation for all combinations that contain $\mathcal{L}\_{\text{guidance}}$, and in particular $\mathcal{L}\_{\text{guidance}}$ alone (and not $\mathcal{L}\_{\text{consistence}}$). In this state, it is not clear what, e.g., $\mathcal{L}\_{\text{cycle}}$ or $\mathcal{L}\_{\text{consistance}}$ contribute separately. - How are the hyper-parameter $\eta$ chosen?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

- A statement on limitations regarding empirical evaluation exist. - Possible negative societal impact, e.g., regarding generation of deepfakes using transfer learning, is acknowledged.

Reviewer 53Zj2024-08-10

Thank you for the response to my review and comments. I believe the method would make good contribution to the community, but a sparse and not revised experimental section prohibit a higher score. Having read the reviews of the other reviewers, I agree that the method clearly needs more evaluations.

Reviewer 9nph7/10 · confidence 4/52024-07-13

Summary

introduces a novel approach called Transfer Guided Diffusion Process (TGDP) for transferring knowledge from a pre-trained diffusion model in the source domain to a target domain with limited data. The authors present the methodology for transferring knowledge, including the formulation of the guidance network and its learning process. They also extend TGDP to a conditional version for joint data and label distribution modeling.

Strengths

1. TGDP offers a new perspective on transferring knowledge from pre-trained diffusion models to new domains with limited data. The whole framework is innovative and reasonable. 2. The paper provides a solid theoretical basis for TGDP, including proofs for the optimality of the approach.

Weaknesses

1. More experiments on real-world datasets are required to further validate the effectiveness of the proposed framework. 2. The computational cost of training the guidance network and the diffusion model should be discussed. 3. The paper primarily validates TGDP on Gaussian mixture simulations and ECG datasets. Its performance on other types of data or domains is not empirically tested or discussed.

Questions

In what scenarios does the proposed framework work best?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors have discussed limitations well.

Reviewer brAM2024-08-11

I thank the authors for their response. I will keep my original rating.

Reviewer 9nph2024-08-12

Thanks for the detailed response. My concerns have been properly addressed and I decide to raise my score to 7.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC