Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models

Thank you for your thoughtful comments and for taking the time to review our paper. We have carefully considered your feedback and would like to provide some clarifications regarding our work. > **1. Computational Advantage** **While a reduction of 1.35 GB in GPU memory may seem modest under standard workstation settings (e.g. A100; 80GB), it represents a significant advantage in the context of on-device training (e.g. IPhone15; 6~8GB).** For training large generative models with on-device chipsets, even a small reduction in GPU memory can have a substantial impact on feasibility. Even an increase in GPU memory of less than 1 GB can destabilize chipset functioning and make computation infeasible. In this context, our approach offers a crucial advantage in on-device fine-tuning of diffusion models by reducing training memory to a level as low as that required for inference, while standard LoRA consumes 35% more memory than Hollowed Net in proportional terms. > **2. Novelty of the Work** **We would like to emphasize that there is a clear distinction between our approach, Hollowed Net, and traditional layer pruning methods.** As presented in recent work [1], accepted at ECCV’24, layer pruning involves completely removing selected layers, which necessitates extensive pre-training with large datasets to recover lost information and restore functionality. Diffusion models, however, often suffer from significant performance degradation post-pruning, as lost information may not be fully recoverable through pre-training. In the table below, we present experiments with layer-pruned Stable Diffusion models by recent work (BK-SDM [1]), accepted at ECCV’24, using rank-128 LoRA. Compared to the results in Table 1 of our paper, these models achieve memory usage comparable to Hollowed Net but show significant performance degradation, generating malformed images for subjects and prompts outside the pre-training data. *Despite the extensive pre-training, the performance of these models is compromised.* | Method | # of Parameters | Model Size | Training Memory | DINO | CLIP-I | CLIP-T | |----------------|------------------|------------|-----------------|---------------|---------------|--------------| | BK-SDM-Base | 595.7M | 2.272GB | 3.546GB | 0.629 ± 0.012 | 0.788 ± 0.007 | 0.3 ± 0.001 | | BK-SDM-SMALL | 496.2M | 1.893GB | 3.133GB | 0.602 ± 0.013 | 0.774 ± 0.008 | 0.298 ± 0.001| **In contrast, Hollowed Net does not completely remove the middle layers and does not require any extensive pre-training.** We temporarily exclude the selected layers during fine-tuning, while preserving essential information from those excluded layers by incorporating an additional buffering stage. Despite this buffering, the computational load for the entire training process of Hollowed Net (2052 TFLOPs) remains more efficient than LoRA fine-tuning (2148 TFLOPs). **Our approach of excluding layers for fine-tuning is a novel technique beyond traditional layer pruning, and merits further discussion within the community.** In fact, Meta AI has recently released an approach for LLMs, shortly before our submission [2], which suggests that for fine-tuning LLM models, one can remove up to 40% of middle layers of the models and perform fine-tuning, and still achieve the comparable results. However, their approach differs from ours due to the characteristics of LLMs versus diffusion U-Nets. Meta AI's method involves complete removal of middle layers for both fine-tuning and inference, arguing that these layers do not store critical knowledge compared to shallow layers. In contrast, our research shows that the deep layers of diffusion U-Nets contain crucial high-level image features, and their complete removal can lead to severe performance degradation, even with additional pre-training (as discussed for the case of [1] above). This underlines the significance of our two-stage fine-tuning strategy, which maintains performance while reducing memory usage without requiring additional pre-training. [1] Kim, Bo-Kyeong, et al. "Bk-sdm: A lightweight, fast, and cheap version of stable diffusion." arXiv preprint arXiv:2305.15798 (2023). [2] Gromov, Andrey, et al. "The unreasonable ineffectiveness of the deeper layers." arXiv preprint arXiv:2403.17887 (2024).

> **3. Discussion on Layer Selection** **Regarding layer selection, we do not base our choices on computational expense of certain layers.** Instead, we leverage the skip connections inherent in the U-Net architecture to determine which layers to remove. For our main results, we choose the third layer of the second down block, the entire third down block, the entire mid block, and the entire first up block to be hollowed during fine-tuning, which corresponds to 40% of the U-Net's parameters. Our findings suggest that removing those layers during fine-tuning is feasible because these deep layers are more involved with high-level abstract features (e.g., the concept of a dog) rather than pixel-level personalization details of a specific instance (e.g., “a [V] dog”), which leads to relatively smaller change of weights in LoRA parameters for personalization in those layers (commonly observed for different models and datasets as presented in the pdf of the top-level comment). This characteristic allows us to use intermediate activations from the frozen original network, acquired with a class-level prompt (e.g., “a dog”) during the collection stage, to update the layers of Hollowed Net during the fine-tuning stage. > **4. Scalability of the Work** **To demonstrate the scalability of Hollowed Net, we present experimental results with CustomConcept101, the largest dataset available for T2I personalization, and results with SDXL, one of the largest publicly available diffusion models (included in the top-level comment and pdf), as well as user studies to validate the consistency of metrics with human evaluation (included in the comment for Reviewer SH8P).** These extensive results consistently show that Hollowed Net *does not require any trade-off* between performance and computation efficiency compared to LoRA, for the removal of up to 40% of parameters. Our approach maintains the same or even slightly better performance compared to LoRA, while achieving significant computational advantages for on-device learning. As our novel two-stage fine-tuning strategy allows to maintain the knowledge of (temporarily) hollowed layers, it enables effective personalization for over 100 subjects with various prompts, including long range image editing like background editing and many others. We hope this detailed explanation addresses your concerns regarding the novelty and effectiveness of our work. We are open to further discussion and clarifications if needed, and we sincerely hope these points will contribute to a reconsideration of the final rating. Thank you for your time and consideration.

Thank you for your kind response. We are sincerely grateful for the time and effort you dedicated to thoroughly reviewing our paper and participating in the discussion. We sincerely hope this additional information addresses your concerns and contribute to a more favorable assessment of our work. We will carefully consider all the feedback you provided during the rebuttal and discussion periods as we update the paper to enhance its completeness. Once again, thank you very much for your thoughtful and thorough review.

Dear Reviewer cT56, As the discussion period draws to a close, we are mindful of the limited time available to address any further questions or requests for clarification. To ensure we provide all necessary information for your final recommendation, we would like to further clarify our previous rebuttal, particularly regarding scalability and generalizability, which we referenced other comments we made but did not directly address. **To address your concerns on the scalability and generalizability of Hollowed Net,** we have presented experimental results with SDXL, one of the largest publicly available diffusion models, and results with CustomConcept101, the largest dataset available for T2I personalization (included in the top-level comment and pdf), as well as user studies to validate the consistency of metrics with human evaluation (included in the comment for Reviewer SH8P). Across all these experiments, Hollowed Net consistently demonstrates its ability to enable effective personalization for over 100 subjects with various challenging prompts of different applications including background editing, artistic rendition, accessorization, and property modification. This performance is comparable to Full / LoRA fine-tuning of diffusion models, while significantly reducing the required training memory. Our findings suggest that Hollowed Net’s approach of excluding middle layers during fine-tuning is feasible because these deep layers are more involved with high-level abstract features (e.g., the concept of a dog) rather than pixel-level personalization details of a specific instance (e.g., “a [V] dog”), which leads to relatively smaller change in LoRA parameters during personalization in those layers, a trend commonly observed across different models and datasets (as presented in the pdf of the top-level comment). This characteristic allows us to use intermediate activations from the frozen original network, obtained with a class-level prompt (e.g., “a dog”) during the collection stage, to update the layers of Hollowed Net during the fine-tuning stage. As elaborated in our rebuttal above, Hollowed Net is a novel method that overcomes the fundamental limitation of fine-tuning methods based on reducing updating parameters, such as SVDiff, which cannot reduce the actual model size that requires a significant amount of GPU memory during fine-tuning. While a reduction of 1.35 GB in GPU memory may seem modest under standard workstation settings (e.g. A100; 80GB), it represents a significant advantage in the context of on-device training (e.g. IPhone15; 6~8GB). For training large generative models with on-device chipsets, even a small reduction in GPU memory can make a substantial impact on feasibility. Even an increase in GPU memory of less than 1 GB can destabilize chipset functioning and make computation infeasible. In this context, our approach offers a crucial advantage in on-device fine-tuning of diffusion models by reducing training memory to a level as low as that required for inference, while standard LoRA consumes 35% more memory than Hollowed Net in proportional terms. We sincerely appreciate the time and effort you have invested in reviewing our work. We hope these clarifications and additional details address your concerns and contribute to a more favorable assessment of our work.

We sincerely appreciate the time and effort you have invested in providing such a detailed review. Your feedback has been instrumental in refining our work, and we are pleased to hear that most of your concerns have been addressed. Regarding the on-device numbers, we regret that time constraints prevented us from conducting additional experiments with actual on-device chipsets. However, we are committed to including a comprehensive analysis regarding on-device numbers in the final version of the paper. In the meantime, we have provided a table below that details the computational loads and memory consumption for each stage of training and inference for both Hollowed Net and LoRA. We hope this offers more clarity, and we will expand on this analysis in the final manuscript (Please refer to the rebuttal comment for Reviewer fGHD for more detailed analysis). | Methods | LoRA | | Hollowed Net | | |---------------|-----------------|--------------------|------------------|--------------------| | | FLOPs | GPU Memory | FLOPs | GPU Memory | | Collection | - | - | 0.238T | 2.138GB | | Fine-tuning | 2.148T | 5.226GB | 2.004T | 3.875GB | | Inference | 0.716T | 3.597GB | 0.920T | 3.588GB | Regarding your additional question, we apologize for any confusion caused by the phrase “using class-level inputs for the middle layers”, which we acknowledge is misleading. Allow us to clarify our assumptions about the potential regularization effect in Hollowed Net. As discussed in [1], fine-tuning a model to replicate a given input image using an instance-level prompt (“a [V] dog”) can lead to language drift and reduced output diversity. To address this issue, [1] introduces class prior preservation loss, a regularizer that trains the model to reproduce a dog image generated with a frozen original U-Net using a class-level prompt (“a dog”). We believe that the architecture of Hollowed Net shares some similar aspects. Specifically, Hollowed Net uses a frozen original U-Net to generate intermediate features with the prompt (“a dog”). These intermediate features are then used as intermediate inputs during the fine-tuning of Hollowed Net. As the model is trained to leverage the class-level features of “a dog” from the original U-Net to generate a specific image for “a [V] dog”, it can retain knowledge about the class prior and integrate it with the subject instance's specific features. This characteristic may explain why Hollowed Net can produce slightly better results than LoRA in some cases. We hope these clarifications and additional details address your concerns and contribute to a more favorable assessment of our work. If you have any further questions or need additional clarification, please let us know. [1] Ruiz, Nataniel, et al. "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

Thank you for considering raising our score. We greatly appreciate your thoughtful feedback and are pleased to hear that most of your concerns have been addressed. Your insights are invaluable, and we are taking them very seriously as we continue to refine our work. To further clarify and resolve your remaining concerns, we would like to provide additional results and explanations regarding the high-fidelity outcomes achieved with a smaller number of samples. When evaluating our method using 200 buffered samples, we achieved the following metrics: DINO: 0.659 (±0.011), CLIP-I: 0.803 (±0.006), and CLIP-T: 0.300 (±0.001). These results are highly comparable to those of Hollowed Net and LoRA FT, as presented in Table 1 of the paper. To explain the minimal effect of using fewer samples, consider collecting 200 samples with a subset of timesteps (e.g., 1, 6, 11, ..., 991, 996). Updating with 200 randomly shuffled samples at a batch size of one would constitute one epoch of 200 steps. Conducting 1000 fine-tuning steps would amount to five epochs. However, it is important to note that in many instances and prompts, 1000 fine-tuning steps are not strictly necessary, as the model often begins to show high-fidelity results after 200-500 steps. Moreover, using a subset of timesteps does not inherently have a negative impact on overall performance. In fact, recent work [1] has shown that sparse updating, utilizing only 1-10% of timesteps, does not degrade performance. In some cases, it even enhances visual quality, presumably due to the regularization effect of sparsity. We believe these considerations contribute to why Hollowed Net's performance remains comparable to LoRA FT even with fewer samples. Besides, please note that Hollowed Net still requires less computational effort for the entire training process than LoRA FT when using up to 600 samples. In our final version, we plan to include extensive ablation studies on the impact of varying the number of samples on both quantitative and qualitative results, similar to the ablation study on fractions of hollowed layers in Section 5.3 of our paper. Furthermore, we will provide a detailed analysis of computational loads and space consumption in the final version, including a breakdown of FLOPs, VRAM, and RAM analysis for different numbers of collected samples and different fractions of hollowed layers, which will enable users to choose the optimal configurations of Hollowed Net according to their specific resource constraints. Once again, we sincerely appreciate your valuable feedback and consideration. We hope this additional information addresses your concerns and aids in your final recommendation. [1] Lu, Haoming, et al. "Specialist diffusion: Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.