How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?

Summary

This paper introduces a novel approach called CIDM to address the challenge of Concept-Incremental Flexible Customization in text-to-image generation. The authors identify two key issues in CIFC: catastrophic forgetting of previously learned concepts and concept neglect during multi-concept composition. The proposed CIDM tackles these problems through several innovative components, including a concept consolidation loss, elastic weight aggregation, and a context-controllable synthesis strategy. The paper provides experiments on various customization tasks to demonstrate the effectiveness of their approach compared to existing methods.

Strengths

1. The paper addresses a practical problem in custom text-to-image generation, namely the ability to continually learn new concepts without forgetting old ones. 2. The ablation studies provide insights into the contribution of each proposed component, strengthening the paper's technical depth. 3. The approach is model-agnostic and has been tested on different backbone architectures (SD-1.5 and SDXL), showing its broad applicability.

Weaknesses

1. The symbol definitions are too complex which are hard to follow. 2. The paper introduces several new hyperparameters. How sensitive is the model to these? 3. How well does the model perform across different application domains and with varied types of personalized concepts?

Questions

See weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

This paper explores a novel and practically significant problem, namely how custom diffusion models can continuously learn new personalized concepts while avoiding catastrophic forgetting and concept neglect. The authors developed a concept consolidation loss and an elastic weight aggregation module to explore task-specific knowledge and task-shared knowledge, and aggregated all low-rank weights of old concepts based on the contributions of old concepts. To address the problem of concept neglect, the authors propose a context-controllable synthesis strategy that can utilize expressive regional features and noise estimation to control the generated contexts. Experimental results confirm the effectiveness of the proposed method.

Strengths

1. The continual learning of generative models is an interesting and meaningful task. How to mitigate catastrophic forgetting and concept neglect in concept-incremental flexible customization scenario remains insufficiently explored. 2. The method proposed by the authors effectively mitigates the challenges of catastrophic forgetting and concept neglect, with detailed descriptions and comparisons in both methodology and analysis. 3. The writing is good and easy to read.

Weaknesses

1. It would be better to mention the comparison methods of experiments in the related work，such as LoRA-M, LoRA-C, CLoRA and L2DM, and emphasize the differences between them and the proposed method. 2. This paper attempts to address two challenges in the Concept-Incremental text-to-image Diffusion Model: catastrophic forgetting and concept neglect. However, from the text, I did not understand the relationship between continual learning and concept neglect. I would like to know if concept neglect is a challenge introduced by continual learning, and what impact continual learning has on original concept neglect? 3. The baseline choices for continual learning such as EWC and LWF are too outdated. Why not choose new and more effective continual learning algorithms? Intuitively, more advanced algorithms should be able to better mitigate catastrophic forgetting. 4. Due to the past focus of continual learning mainly on classification tasks, I hope to see more discussion on the background of continual generation models. For example, in this setting, how severe are catastrophic forgetting and concept neglect specifically, and whether some more objective evaluation criteria can be introduced. 5. How much do different weighting schemes between LoRAs affect the quality of generation, compared to some intuitive weighting baselines? 6. Does the value of the rank in LoRA and its placement in the model (e.g., which layer) have an impact on the results?

Questions

Please check the above concerns in the weakness points. And If my concerns are resolved, I will consider increasing my score.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

See the weaknesses above.

Summary

This paper introduces the Concept-Incremental text-to-image Diffusion Model (CIDM), which addresses the Concept-Incremental Flexible Customization (CIFC) problem. This approach represents one of the first explorations into learning new customization tasks incrementally, effectively navigating the dual challenges of catastrophic forgetting and concept neglect. To combat catastrophic forgetting, the paper proposes a new concept consolidation loss coupled with an elastic weight aggregation module, which together helps preserve knowledge of previously learned personalized concepts. Additionally, the development of a novel context-controllable synthesis strategy specifically addresses the challenge of concept neglect, ensuring that new concepts are integrated without overshadowing existing knowledge. The effectiveness of the CIDM is validated through comprehensive experiments across a range of text-to-image generation tasks, demonstrating its capability to learn new customization tasks consecutively with notable efficacy.

Strengths

1. The paper commendably defines the Concept-Incremental Flexible Customization problem, setting a precedent for addressing this emerging challenge within the field. CIFC is a practical issue crucial for continuously synthesizing personalized concepts based on user preferences in real-world applications. 2. The novel model introduced by the authors incorporates a range of innovative functions and algorithms, which are thoroughly explained and convincingly motivated. The concept consolidation loss and the elastic weight aggregation module are noteworthy among these, paired with a context-controllable synthesis strategy. These elements collectively aim to mitigate issues of catastrophic forgetting and concept neglect effectively. 3. The experimental section of the paper is detailed, providing a broad comparison of the model’s capabilities across tasks such as multi-concept generation, style transfer, and image editing.

Weaknesses

1. A critical concern arises regarding the scalability of the model, specifically the accumulated memory load after learning each task. This is particularly pertinent in continual learning scenarios where efficiency is crucial. 2. The paper lacks detailed implementation information on critical tasks such as custom image editing and style transfer. Providing these details would significantly enhance the clarity and replicability of the research. 3. The concept of balancing task-specific and task-shared knowledge is intriguing and seems to be central to effectively addressing the CIFC problem. To substantiate the claims of its effectiveness, the authors should include ablation studies that isolate and quantify the impact of these components. Additionally, visualizations comparing the results with and without these strategies would provide a clearer, more direct illustration of their value.

Questions

1. I recommend the authors include details on memory consumption or the number of stored parameters after each task during the rebuttal process. 2. The authors should incorporate a comprehensive description of custom image editing and style transfer processes, including specific parameters and settings used, to allow for a better understanding and evaluation of the proposed methods' effectiveness in practical applications.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

There are no potential negative societal impacts.

Summary

This paper tackles the problem of continually adapting text-to-image diffusion customization models. The proposed method employs a novel concept consolidation loss, elastic weight aggregation module, and context-controllable synthesis strategy. Extensive experiments demonstrate that the proposed method performs strongly compared to SOTA baseline methods.

Strengths

1. The approach is innovative and presented in a principled manner. 2. The problem setting is a strong area of impact for research with real-world applications, which is important for continual learning. 3. The experiments and analysis are very extensive, and the paper presentation is of high quality.

Weaknesses

1. Please include a table that compares the training, parameter, and memory costs of your method against other methods. This comparison is crucial for a comprehensive evaluation of your method. 2. Although the results are clear and detailed, they lack confidence intervals or similar statistical measures in the quantitative metrics. Running the experiments with multiple random seeds would ensure robustness and provide a more reliable assessment. 3. The dataset used is quite limited. To fully justify your setting, you should explore more diverse datasets and longer task sequences. For instance, with only 10 simple concepts, it would be feasible to store a LoRA version of Custom Diffusion or even the Custom Diffusion deltas in memory and merge them on the fly for the target task, achieving zero forgetting in a practical manner. To truly demonstrate the impact of your method and setting, beyond what can be trivially solved by adapters or state-of-the-art LoRA-based merging methods, you should consider long task sequences or challenging benchmarks such as celebrity faces.

Questions

Please see my weaknesses section. I think 1 and 2 are crucial for acceptance (would raise my score to WA if done thoroughly). Part 3 is important, too, but more difficult to address in a short period of time.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Thank you for your generous support!

Dear Reviewer YpEo, Glad to hear that your concerns have been addressed well. Thank you for your great efforts in reviewing and for the good questions. Best regards, Authors of Paper #1919