DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models

Summary

This paper proposes DreamSteerer, a plug-in method that enhances existing text-to-image personalization techniques by improving the editability of source images. DreamSteerer proposes a novel Editability Driven Score Distillation (EDSD) objective to improve the structural alignment between the source and edited image by performing score distillation with respect to personalized model parameters. It identifies and addresses mode-trapping issues in EDSD using regularization with spatial feature-guided sampling. The proposed method enhances the editability of several T2I personalization methods. Moreover, they introduce two key modifications to the Delta Denoising Score framework to enable high-fidelity local editing with personalized concepts.

Strengths

* The problem of enhancing the editability of personalized image editing conditioned on source images is significant and underexplored. This idea is original since it modifies the score distillation methods to better align with the personalized image editing task. They provide an easy-to-use model-agnostic method for any type of personalization. * The quantitative experiment results are solid and demonstrate the method's effectiveness over the three baselines. * The presentation and writing are reasonably clear and easy to follow.

Weaknesses

* The method lacks comparisons with editing baselines outside of personalization. It also lacks qualitative or quantitative comparison to the cited preliminary works [8, 14]. For the images that also appear in the Custom-Edit [8] such as rows 1,2,5 in Figure 1 of this paper, the result images in the Custom-Edit [8] paper appear to have much better quality. * The "mode shifting regularization" approach is not convincing on the effectiveness of the proposed(Figure 5). In Fig.5, the claimed improvement in editing fidelity from Fig.5 (c) to Fig. 5 (d) does not seem obvious. (also between "(f)" and "(g)" ) * The paper is missing some important metrics for evaluation. In Section 5, the paper finds that automatic metrics do not fully reflect the superior performance of the proposed method. There is no image quality assessment (IQA) metrics used such as [ref1, ref2, ref3]. Also, the structural similarity metrics globally compare the edited image to the source image, which is unsuitable in this scenario since you want to maintain the similarity in the unedited background region, and a local similarity comparison is better suited. [Please check Questions] * The method is claimed to be efficient in the abstract section. However, the proposed method's speed and cost are not analyzed or numerically measured. The method appears to require significant computation for a single edit. It is recommended that the computational overhead be reported compared to base personalization methods like DreamBooth. * The example images in the paper have very low resolution, especially when zoomed in.

Questions

* In the Method section, how is the given personalized model trained, and on which data? * In Figure 2, what are the reference images? Are they the same as those in Figure 3? * In Table 1, what do the user preference numbers indicate? * What is the computational overhead compared to a base personalization method like DreamBooth? How many UNet calls are required to edit one single image? * How would the performance of the proposed method compare with the recent personalization methods like DreamMatcher? * The paper is missing some important recent works in personalization: * DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization (CVPR 2024) * Style Aligned Image Generation via Shared Attention (CVPR 2024) * Visual Instruction Inversion: Image Editing via Visual Prompting (NeurIPS 2023) * MagicFusion: Boosting Text-to-Image Generation Performance by Fusing Diffusion Models * References for Weakness section: * [ref1] C. Chen, J. Mo, J. Hou, H. Wu, L. Liao, W. Sun, Q. Yan, and W. Lin. Topiq: A top-down approach from semantics to distortions for image quality assessment. IEEE Transactions on Image Processing, 2024. * [ref2] J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, 2021. * [ref3] W. Zhang, G. Zhai, Y. Wei, X. Yang, and K. Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

Discussed in Appendix J

Summary

The paper introduces DreamSteerer, a method to enhance the editability of source images using personalized diffusion models in text-to-image (T2I) personalization. Existing methods often fail to maintain edit fidelity when applied to new contexts due to limited reference images and adaptability issues. DreamSteerer addresses this by introducing the Editability Driven Score Distillation (EDSD) objective, which enhances image editability. To mitigate a mode trapping issue in EDSD, the paper proposes mode shifting regularization with spatial feature guided sampling. This technique aligns the model more closely with the source image structure while preserving personalized concept appearances. Extensive experiments show that DreamSteerer significantly improves editability and efficiency across various T2I personalization baselines.

Strengths

1、The method is designed as a plug-in compatible with arbitrary personalization baselines. This versatility enhances its applicability across various models and use cases 2、DreamSteerer requires only a small number of fine-tuning steps (~10) to achieve significant improvements, making it computationally efficient.

Weaknesses

1、While the paper presents a novel approach, it would benefit from a more thorough discussion regarding its innovative aspects in comparison to existing work in Object Driven Image Editing, such as DreamEdit[1] and SWAPANYTHING[2]. It appears that similar mechanisms are already in place within these frameworks, and a detailed exploration of how the current method distinguishes itself or builds upon these concepts would be valuable. 2、A broader experimental evaluation is suggested. This should include comparative studies with existing image editing works and other Object Driven Image Editing techniques mentioned in [1,2]. 3、The proposed method relies on certain heuristics, such as spatial feature guided sampling and the choice of specific parameters for mode shifting regularization. These heuristics may not generalize well across different datasets or tasks, leading to suboptimal performance in some scenarios. [1] DreamEdit: Subject-driven Image Editing [2] SWAPANYTHING: Enabling Arbitrary Object Swapping in Personalized Visual Editing

Questions

Figure 14 and 13 is same. Insufficient resolution of figures provided in the paper

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper discussed the limitations of this work.

Summary

Aiming at addressing unsatisfactory editability on the source image, this paper proposes a novel plug-in method for augmenting existing T2I personalization methods. Specifically, this framework finetunes the personalization parameters by training a novel Editability Driven Score Distillation objective under the constraint of a Mode Shifting regularization term based on spatial feature-guided samples. Extensive experiments validate the effectiveness of the proposed method.

Strengths

1. The proposed method can be applied current T2I personalization models to perform custom editing as a plug-in. 2. Efficiently, the framework DreamSteerer requires only a small number of fine-tuning steps (∼ 10) to achieve significant improvement in editability on a source image.

Weaknesses

1. In related work, the authors lack a discussion of recent works for T2I personalization. Most of the works are in the first half of last year. 2. This framework introduces many computing loads in there modules in Fig.3, which worries me about the inference efficiency of this method. It is suggested that the authors can provide some explanation about computational complexity in experiment section. 3. There are some excellent methods for custom editing, such as [1] [2]. It is recommended to compared with these strong baseline to show the effectiveness of this method. 4. As for qualitative results in this paper, obviously, the proposed method forces consistency of the object shape between the source object and target object, resulting in the destruction of the edited object identity compared the reference. This may not be the editing result that the user wants. It is suggested the authors can provide analysis for this. [1] Custom-edit: Text-guided image editing with customized diffusion models. [2] DreamEdit: Subject-driven Image Editing.

Questions

See weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Although DreamSteerer achieves high-fidelity editing results, its performance is still limited by the baseline model.

Summary

DreamSteerer is a proposed fine-tuning pipeline for personalized diffusion models, designed to enhance the custom editability of these models. The authors point out that naively incorporating existing image editing and personalization methods—such as employing score distillation and DDS where the differentiable function is initialized by the source image latent space—results in low fidelity and lack of editability relative to the source image. To address these issues, they propose the following additions that comprise DreamSteerer: 1. Editability driven score distillation (EDSD), to increase the editability of the model by distillation information from the source model $\epsilon_{\phi_0}$ to the personalized diffusion model $\epsilon_{\phi_0}$ through a single-step perturbed source latent state 2. Spatial Feature Guided Sampling and mode shifting regularization to overcome the mode trapping issue introduced by EDSD (commonly observed in score distillation) 3. Source score bias correction and automatic subject masking to address the distribution shift from personalization, and to focus on the relevant aspects of the image during training respectively. The authors present thorough qualitative and quantitative results across three metrics: semantic similarity, perceptual similarity, and structural similarity, along with a human preference evaluation. They demonstrate that DreamSteerer consistently improves upon existing baseline methods and include ablation studies to justify various components of the pipeline.

Strengths

Originality: The work is well-motivated and addresses a limitation in the intersection of personalization and image editing, which is a relatively underexplored area. Although DDS has been employed for general text-driven editing, the application of DDS to enhancing editability in the personalization setting is a novel approach. In addition to combining the existing ideas, the authors introduce further modifications to improve final results. Clarity: The paper clearly explains each learning objective with detailed formulations, supported by an overarching diagram. The results are well-organized and articulated, presenting both qualitative and quantitative analyses. This structure provides a cohesive narrative that justifies the design decisions made throughout the work. Quality: The proposed algorithm introduces a novel approach which improves over existing methods in the personalized image editing domain, and the work presents sufficient ablation studies over core components to reinforce the reliability of the proposed approach. Significance: The proposed method for personalized image editing with few fine-tuning steps required has potential for widespread adoption. DreamSteerer's ability to maintain structural integrity while enabling personalization edits makes the work applicable across various creative domains, which may demand precise edits.

Weaknesses

The paper’s results would benefit from benchmarking on more extensive personalization datasets, such as CustomConcept101. Currently, the evaluation is based on only 16 concepts from the ViCo repository, a design decision that lacks strong justification. While the selected concepts show a reasonable level of diversity, evaluating the method across a broader set of personalized concepts would enhance the robustness and generalizability of the results. Additionally, there are minor implementation details that could be clarified in the Experiments section. Specifically, the paper mentions using pre-trained checkpoints provided by ViCo. However, the ViCo GitHub repository does not link to checkpoints for Textual Inversion, DreamBooth, and Custom Diffusion, leading to some confusion about the source of these pre-trained checkpoints. Providing clear information on where these checkpoints were obtained would help improve the reproducibility of the work.

Questions

I have minor questions regarding the implementation details to clarify some confusion: 1. Source of Real-World Images: Can you provide more details on the 70 random real-world images used for editing? Were these images scraped from the web or sourced from an existing dataset? Additionally, were images containing a shared superclass used to enhance the feasibility of the editing tasks? 2. Clarification on Figure 5: In Figure 5, it’s unclear where the results in subfigures (e) to (g) originate from. Are these images based on specific source images, or are they entirely generated without any reference images, from the finetuned personalization model?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have adequately addressed the limitations and potential societal impact of their work. Specifically, they describe and visualize the limitations of DreamSteerer, which can be constrained by the baseline personalization model, and suggest ways to avoid fidelity inconsistencies in future similar works.

[Gentle reminder] One day remains before the end of discussion period

Dear reviewers and AC, Thank you for the time and effort you have invested in reviewing our manuscript and providing insightful feedback. We have carefully considered each of your comments and have prepared detailed responses to address your questions and concerns. It is our hope that these responses have satisfactorily resolved your concerns. As we near the end of the discussion period, we welcome any additional questions or feedback you may have. Warm regards, Authors

comment from reviewer

Thank you for answering my questions. After reading the author's rebuttal, I think the current version still needs revision in terms of (I) experiments on the sensitivity of the heuristic network to hyperparameters (the specific value of the hyperparameter λ is also missing in the implementation details), (ii) a more significant performance improvement compared to Custom-Edit, and (iii) further revisions to the current paper, such as supplementing the paper with additional details like λ and also the use of lossless compression to reduce the volume while maintaining the sharpness of the image. Therefore, I will keep my rating same.