Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics

Summary

Text-to-image generation models offer fine-grained control over synthesized images, but fast adaptation to smaller datasets or new concepts remains a challenge. Existing efficient adaptation methods suffer from long training times, hindering practical applications and resource usage. This work addresses the issue by studying the training dynamics of popular text-to-image personalization methods and proposes a drop-in early stopping criterion that significantly speeds up adaptation (up to 8 times faster) without compromising quality, as demonstrated through experiments on Stable Diffusion and various personalization methods.

Strengths

1. This paper proposes a simple but effective method to accelerate text-to-image customization. 2. The proposed method is well-motivated and easy to understand.

Weaknesses

1. It is imperative to provide supporting evidence to justify the necessity of adaptive step choices. Can we simply set a fixed step number (e.g. reduce to 1/3) without losing much performance? For instance, analyzing the outcomes and plotting the distribution of selected step numbers can demonstrate the potential reduction in unnecessary iterations. This approach would enhance the validity of the proposed method. 2. The authors should engage in a thorough discussion of pertinent literature concerning the acceleration of generative models. It is crucial to acknowledge and reference closely related research in this domain. --- Having read the author's rebuttal, I've chosen not to alter my score.

Questions

Please address the issues highlighted in the Weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Please address the issues highlighted in the Weaknesses section.

Summary

This paper studies the training dynamics of a few state-of-the-art txt-to-image personalization methods and proposes an early stopping criteria while fine-tuning the base models to speedup their customization. They evaluate this criteria on Dreambooth, Textual inversion, and custom diffusion on 18 concepts and showing close or slightly better img-txt clip similarity scores on the validation prompts while degrading the similarity to the input image.

Strengths

1. This paper presents a comprehensive study on the randomness factors during training of the customization models and their training curves and propose a simple stopping criteria based on a deterministic variance evaluation of the diffusion loss. 2. This early stopping results in a 2x speedup on custom diffusion as well as dreambooth-LoRA and 12x speedup in textual inversion.

Weaknesses

1. The whole idea of the paper is not very novel, and looks more like an analysis paper. 1. There is a trade-off in the samples similarity to the source image during both training and validation versus the number of finetuning steps. Although the authors have shown more generalization ability to validation prompts, it looks like identity preservation has been degraded. 2. The evaluations are only done on 18 concept examples. A larger set would be needed to make the results more convincing. 3. The scores which are reported in Table 2, only measure the similarity of the generated image with the validation text prompt. It would be important to see how much of the identity is preserved in these generations using the proposed early stopping criteria. Identity preservation is an important goal in personalization methods and is not studied well in this paper.

Questions

1. In addition to the above questions mentioned above, it was not clear to me if the models are trained on all concepts together or if they are trained on the samples of each concept. If it is on each concept, where are 8 images per batch coming from? If that has been done on multi-concepts, then I'd expect to see some evaluations on multi-concept customization. 2. Very few qualitative examples are shown in the paper/supplemental. It would be interesting to add more examples comparing the results with the baseline.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

- The proposed technique causes a natural trade-off between the number of fine-tuning iterations and identity preservation of the objects in the source images. While the latter is quite important in image personalization, it is degraded in this paper (based on Table 1) and is not studied well in human evaluations.

Summary

This paper argues that customization techniques for diffusion-based text-to-image generation models train for longer than is needed. This is because the training loss of diffusion models is often not informative -- i.e., often looks like stationary noise -- so practitioners tend to use a fixed (often excessive) number of training steps. This paper identifies and analyzes the sources of stochasticity in the training loss, and propose simple ways to eliminate them to make the loss more informative. They also introduce a simple early stopping criteria based on this interpretable loss.

Strengths

Paper addresses a key issue many people training diffusion models face: the loss is not informative. That is, it often behaves like stationary noise despite the model continuing to improve on auxiliary metrics of interest (FID, human evals). The authors do a very principled analysis of the sources of stochasticity in this loss and identify the sampled time-step to be the main driver. Eliminating this randomness leads to loss that better corresponds to model performance. I could se this becoming common practice -- not just in the model customization/fine-tuning regime but also in the training of the base diffusion model.

Weaknesses

The main weakness of the paper is that I'm not sure how useful the DVAR early stopping criteria is. See question below. However for this paper, this is less of an issue, since it's main contribution is a careful analysis of a problem people who train diffusion models face: uninformative loss and where it comes from.

Questions

How does L_det vary across concepts/models/methods? One plot that could help motivate the DVAR is a plot that shows how L_det varies across concepts (and models/methods). If it's the case that L_det doesn't vary very much then maybe picking a fixed (but smaller) number of fine-tuning steps is sufficient.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

n/a

Summary

This paper studies the training dynamics of popular text-to-image personalization methods (such as Textual Inversion, DreamBooth, and Costom Diffsuion), aiming to speed them up by an early stopping approach which allows the model to optimize or fine-tune for fewer iterations. A key observation is that most concepts are learned at early stages and do not improve in quality later, but standard model convergence metrics fail to indicate that. Based on this observation, the authors propose a simple drop-in early stopping criterion that only requires computing the regular training objective on a fixed set of inputs for all training iterations. Experiments are conducted on Textual Inversion, DreamBooth, and Costom Diffusion.

Strengths

1. The observation that most concepts are learned at early stages and do not improve in quality later, but standard model convergence metrics fail to indicate that is interesting and inspiring. 2. The proposed approach is well-motivated with key observations and in-depth analysis before deriving the method. 3. The proposed approach improves the efficiency of personalized text-to-image models by a simple but effective early stopping scheme.

Weaknesses

1. Although the visual results and CLIP scores indicate no further improvement after optimization for a certain number of steps, it is still not clear whether the observation is solid. The CLIP score can be biased because of the limited ability of CLIP in understanding complex and detailed information. Good visual results in generating an image similar to the input image do not mean perfect identity and detail perservation for personalized text-to-image generation. 2. The writing and logic can be improved. For example, how does 3.2 (investigating the sources of randomness) relate to other sections is not well demonstrated. 3. The evaluation is limited. Firstly, only 18 concepts are used for evaluation. Prior work such as DreamBooth actually used more concepts and prompts. Secondly, the CLIP image-image and CLIP image-text similarities are not enough to evaluate the concept preservation and image quality of personalized diffusion models. Prior work DreamBooth used several other evaluation metrics to reflect the ability of the models. Thirdly, the visual results in Figure 4 are not impressive, and there are no examples for personalized generation where the user provides a different text prompt with the same visual concept from the original image but in a different background or scene. The different application scenarios of DreamBooth, Textual Inversion, and Custom Diffusion are not extensively experimented.

Questions

Please refer to the weakness section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors addressed the limitations.

revised rating

Thank the authors for the response. The authors have addressed my concerns about the limited concepts and metrics used for evaluation by adding more concepts and evaluation metrics in the rebuttal. The authors also explained that they showed the personalized generation results in the paper although it was not illustrated properly (prompts are not shown so it was difficult to understand what the images mean). This can be improved in the final version of the paper. My remaining concern is that the proposed approach does not seem to demonstrate good identity and detail preservation as shown in the examples in the paper and the additional rebuttal page. Therefore, I increase my rating to borderline reject.

Revised rating

Thanks for the rebuttal. Since most of my concerns are addressed in this rebuttal, I'd be happy to increase my rating.