Controlling Text-to-Image Diffusion by Orthogonal Finetuning

Summary

This paper proposes a parameter-efficient fine-tuning algorithm that preserves the so-called "hyperspherical energy" to achieve a great trade-off between prevervation and generalization and high training efficiency. The authors conduct experiments on subject-driven generation and controllable generation, which both can improve the image generation quality as compared with other SOTA methods.

Strengths

1. This paper presents the orthogonal finetuning idea clearly. It provides two toy examples to help readers understand the motivation of this paper. 2. The core idea of orthogonal finetuning is straightforward and easy to understand. The authors also cite some theoretical papers about "hyperspherical energy" to support their motivation and algorithmic foundation. 3. This paper provides another perspective for the field of PEFT other than LoRA, which may excite more interesting works and discussions on this important topic. 4. The authors conduct two experiments: subject-driven generation and controllable generation based on Dreambooth and Controlnet, both of which demonstrate their effectiveness.

Weaknesses

1. The "subject-driven generation" introduced in related work is inadequate. Most works you mentioned are about image-editing and some recent dreambooth-like subject-driven finetuing works are not included, such as P+, DisenBooth, Cones Voynov, Andrey, et al. "$ P+ $: Extended Textual Conditioning in Text-to-Image Generation." arXiv preprint arXiv:2303.09522 (2023). Liu, Zhiheng, et al. "Cones: Concept neurons in diffusion models for customized generation." arXiv preprint arXiv:2303.05125 (2023). Chen, Hong, et al. "DisenBooth: Disentangled Parameter-Efficient Tuning for Subject-Driven Text-to-Image Generation." arXiv preprint arXiv:2305.03374 (2023). Although this work may be close to your submission time, you'd better add some discussion or comparisons with this kind of work. 2. In Table 1 and Table 2, the statistical difference between COFT and OFT is very marginal; how can you draw the conclusion that COFT is better than OFT? 3. In the experiment section, no specific-domain finetuning comparisons between OFT and LoRA  are provided. Since LoRA is widely used in many downstream specific-domain finetuning, I'd like to see some further comparisons and discussion.

Questions

1. In Sec. 3.1, you say "orthogonal transformation is a sweet spot between flexibility and regularity." How can you draw the conclusion? lack some theoretical or experimental proof. 2. Figure 3 is the "controllable generation experiment using the setting of ControlNet". The results of controlnet without orthogonal finetuning (the original controlnet training ?) seem to collapse totally. Is it reasonable? 3. In line 197, I don't understand why you define HE(W) as this equation. Any reference? 4. In line 218, how to ensure that Q is a skew-symmetric matrix during code implementation? There is no explanation here or in the appendix.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

This paper seems no conclusion and limitation section.

Summary

This work proposes Orthogonal Finetuning (OFT), a new finetuning method for t2i diffusion models, centered on maintaining the hyperspherical energy from the pretrained model during finetuning to preserve semantic structures. It also introduces Constrained Orthogonal Finetuning (COFT), a variant that limits angular deviation from the pretrained model, thus preserving generative performance from pretraining. OFT is different from LoRA, since it does not assume a low-rank structure for weight updates and instead udpates weights using layer-shared orthogonal transformations. OFT shows good results for subject-driven and controllable generation tasks.

Strengths

1. From my experience, neuron angle is indeed very important and it's interesting that this paper brings this to light with rigorous arguments and some nice toy experiments. 2. The idea of OFT is solid and well-motivated. But there are questions as to the effectiveness compared to other less-well motivated alternatives such as LoRA. 3. I do think the exploration of this idea is worthwhile and this is the reason that I am still voting positively, even given my reservations on evaluation. 4. Supp. material is solid with many more examples and some good nuggets of insight (e.g. tuning conv layers is not very impactful).

Weaknesses

1. DB does not add additional inference overhead (L. 293) 2. Although the idea makes sense, and is an interesting way to see the problem, it is hard to know whether the empirical improvements are enough. Specifically, it seems like some metrics are better (subject fidelity, quality) and some are worse (prompt fidelity, by very little, could be negligible). But there is no user study, which could put these results in perspective. These metrics are not nearly as strong as a user study and a user study could show the real benefits of the method. This is my main reservation with regards to this work. 3. My second reservation is that, for low ranks, it would be great to have comparisons with LoRA (w/ equal number of parameters, since OFT is more parameter efficient at equal rank)

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Societal impact seems to be missing, although is an extension of other personalization methods such as Dreambooth etc. Failure cases would be important for this paper - specifically, when does OFT do less well than LoRA or DB?

Summary

The paper proposes a method for efficiently fine-tuning pre-trained text-to-image generation models, good results are obtained in experiments.

Strengths

The paper proposes OFT and COFT, which can efficiently fine-tune the pre-trained text-to-image generation model as less parameters needed to be fine-tuned compared to baseline methods. Given the fact that text-to-image generation models normally requires massive training samples and computation resources to train, an efficient fine-tuning method could benefit many researchers. According to the experiments, the proposed methods lead to better performance on different tasks.

Weaknesses

Currently, the experiments are not convincing enough. For the proposed methods, one of the most important downstream applications is adapting pre-trained models to a target domain/dataset. Different from customizing with few samples, fine-tuning on downstream dataset can be easily evaluated by some commonly used metrics like FID, KID. Comparing supervised FID on a given dataset, for instance MS-COCO, of proposed methods, LoRA, full parameter fine-tuning is an important experiment. Customize a pre-trained model with few images is an important downstream task, while it is actually very hard to evaluate the performance of different methods. Evaluating with pre-trained models (e.g. CLIP) may not be convincing enough because the novel concepts contained in new images could be outliers. It is actually hard to judge whether identity of target concept is well preserved. Thus human evaluation is suggested to be added. What's more, the method should be able to generate images of different styles (pencil drawing, oil painting), which are shown in related papers, these results are also suggested to be added in the paper. Some experimental details are not clear enough (please see some questions below).

Questions

1. What is the $\theta$ in $z=cos(\theta)$ in Figure 2 of section 3.1? 2. Can the authors provide reference for the statement finetuning essentially aims to train the model while implicitly minimizing $\Vert M - M^0\Vert $? 3. Can the authors provide fine-tuning time in seconds/minutes of different methods, and comparison of sampling time with same number of steps at testing? 4. In Figure 5, there are many details lost in the given examples. How will the results be if using a much larger $\epsilon$? For this example, can the author add results of OFT and DreamBooth (which is provided in their paper Appendix) into Figure 5?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No

Summary

The paper introduces a principled finetuning method called Orthogonal Finetuning (OFT) for adapting text-to-image diffusion models to different downstream tasks. The key idea of the proposed approach is to preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. The paper claims that this property is crucial for preserving the semantic generation ability of T2I models. The paper additionally proposes Constrained Orthogonal Finetuning (COFT) to improve finetuning stability. Experiments on subject-driven generation and controllable generation demonstrate the utility of the proposed approach.

Strengths

The key idea of the paper to learn a layer-shared orthogonal transformation for neurons such that their pairwise angles are preserved is interesting and well-presented. The proposed finetuning approach could be potentially useful/applied to other domains/tasks. The experiment results on subject-driven generation and controllable generation are solid with sufficient comparison to the baselines.

Weaknesses

I do not find any major weaknesses with this work except for few clarification questions: (a) On OFT in section 3.2 and COFT in section 3.4: Can they help learning weight matrices that are identity matrices? From equation (4), it seems COFT can learn this. Did you perform any analysis on the weights to see if the weight matrix can be identity? Would be great if you provide some details in the rebuttal. (b) In section 5.1, it is not clear if the paper applies OFT and COFT to the CLIP Text encoder as well?

Questions

Please see weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors adequately addressed the limitations.

Summary

This paper proposes a new method for finetuning text-to-image diffusion models for better guidance and control. The proposed method OFT is a low-cost finetuning method even in comparison with LoRA, and it can preserve semantic generation ability well. Furthermore, this work proposes COFT to improve the stability of OFT. Experimental results demonstrate impressive performance of the finetuned model on subject driven generation and controllable generation.

Strengths

1. The proposed method is light-weight but yield highly stable finetuning performance in both subject driven generation and controllable generation. 2. The method in simple and well correspond to the motivation. This paper explains the motivation about orthogonal transformation well.

Weaknesses

1. Unfortunately, this paper is incomplete, which does not consist of further discussion and conclusion. 2. In comparison with Dreambooth, the advantages are relatively small.

Questions

1. Is this method generalizable to other models, say large language models?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors did not discuss the limitations and introduce their future work.

Feedback to authors

Thanks for the comment. From the supplementary experimental results I find that this method also works well in other domains and tasks. It also has advantages over Dreambooth. I do agree to level up the score to 6.

Sorry for late response

Dear authors, Sorry for my late response. I have carefully read your rebuttals and discussions with other reviewers. Thank you for addressing most of my questions. Please add your conclusion and limitations to the revised version. The paper is worthy of acceptance. I will keep my score.