Object-aware Inversion and Reassembly for Image Editing

Summary

This paper achieves high-quality object-level image editing through a simple yet effective method. Authors find that the editing for different parts requires different noising levels used to inverse the given image to latent space. As such, they search for optimal inversion steps for different parts based on a search metric for both the edited and non-edited regions respectively. In order to produce a natural final result, the results of different parts are seamlessly blended, which yields a harmonic and globally coherent editing output. Albeit its simplicity, the proposed method outperforms competitive works when editing real images and shows promise in real usage.

Strengths

- This paper identifies an interesting and useful phenomenon that the structure of different objects corresponds to a different level of latent as modeled in the diffusion process. To achieve the optimal trade-off between semantic preservation and text-based editing, the optimal number of inversion steps can be determined based on a quantitative measure. - The image editing results are indeed impressive, as shown in the main text and the appendix. Both the quantitative and qualitative results demonstrate the advantage over strong prior works. - Since the denoising process of different image parts is disentangled, the proposed method supports fine-grained multiple object editing, which is not featured in prior works. - The parallel denoising strategy effectively reduces the search speed.

Weaknesses

- One major drawback of the method is the search speed. Since the inversion steps of each editing part should be comprehensively searched, this may pose a challenge when editing for multiple regions. It would be interesting to conduct a coarse-to-fine search strategy for speedup. Moreover, in the paper, it is suggested to report the image editing speed for different methods since some other baselines, like prompt-to-prompt, can edit images in a feed-forward manner. - Some writing parts can be improved for better clarity. For example, what's the detailed formulation of min-max normalization? It is suggested to better rephrase the term "concept mismatch". - Since the re-inversion step is always set to 20% of the total steps, the non-edit regions will inevitably be affected. Such subtle change is particularly apparent for faces and small objects.

Questions

The proposed method determines the starting point in the latent space to deviate the semantics of the objects. I think this search strategy is orthogonal to other editing methods that manipulate the latent based on the textural prompt. Hence, I wonder whether the proposed method is compatible with other editing methods, like null inversion. I would like to hear the authors' feedback regarding this.

Rating

8: accept, good paper

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Summary

This paper introduces advancements in text-driven image editing using diffusion-based methods. Existing techniques involve a fixed number of inversion steps to edit images aligned with a target prompt, but the optimal number of steps varies for different editing pairs. The paper introduces a new approach called Object-aware Inversion and Reassembly (OIR) to enable fine-grained, object-level editing. The object-level fine-grained editing is achieved by segmentation from SAM. It uses a search metric to determine the optimal inversion step for each editing pair and combines them for the final edited image, demonstrating superior performance, especially in multi-object editing scenarios.

Strengths

1. The paper is well written, and the idea is clearly demonstrated through figures. 2. Although it is natural to do some grid search on the inversion steps when doing image editing, this paper introduces a systematic and principled way to do the search with a quantitative metric. 3. The idea of working on each object separately and reassemble is a interesting way to do multi-object editing. 4. Results are competitive compared to other inversion based editing techniques.

Weaknesses

1. The method introduces significant computational overhead. For each input image, it has to run a search for optimal denoising steps for each edit pair. The search process can be very time consuming. as it requires denoising from each inversion step, plus a metric calculation step. The overall amount of computation for a single image editing task can thus be very large. 2. It relies on SAM to do fine-grained segmentation of the input image to localize the object, and then do localized editing followed by resembling. However, it has some limitations. For example, the segmentation may not work well for small object. In addition, using SAM constrained the edit to object-level, while the applicability of more global change is questionable (e.g., change the style of an image from spring to winter, change the background to Mars, etc.). It may not be able to change the location of an object. 3. Related to above point, the competitive method, such as null-text inversion, does not involve segmentation and localization, therefore it is a bit unfair to compare with them, as in principle they can only benefit from the segmentation and localization. This is independent from the main innovation of this paper, which is searching for optimal inversion step. 4. Overall, it is a method that combines multiple existing component, with significant increase in compute. Therefore, the contribution and significance of the method is limited.

Questions

N/A

Rating

3: reject, not good enough

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

This paper discovers that various editing pairs exhibit differing levels of editing complexity. Additionally, it is observed that neglecting the varying difficulty levels of different editing pairs in multi-object editing tasks results in the problems of concept mismatch and poor editing. To address these issues, the paper introduces a novel training-free image editing approach called OIR. This method follows the approach of assembly first and then reassembly. 1) In the assembly strategy, the paper introduces a novel search metric. This metric automatically identifies the optimal inversion step for different editing pairs, enabling automatic control of editing difficulty. This approach allows different editing pairs to undergo separate denoising, preventing concept mismatch issues. Moreover, employing the search metric to find the optimal result represents a new paradigm in single-object editing. 2) In the reassembly strategy, the article suggests merging the editing regions and non-editing region during the reassembly step. This operation takes place in the denoise latent space. The reassembly process incorporates a re-inversion strategy, enhancing the image editing's edge smoothness, improving image editability, and enabling interaction across regions. 3) To assess OIR's capabilities, the authors collect two datasets, which are employed to evaluate both the single-object editing proficiency of the search metric and the multi-object editing capability of OIR. Numerous experimental results indicate that the search metric performs on par with existing state-of-the-art editing methods in single-object editing tasks. Moreover, in multi-object editing tasks, OIR demonstrates strong performance, outperforming the previous SOTA methods

Strengths

1. This paper identifies a fundamental challenge in multi-object image editing tasks. Previous methods typically treat an entire image as a whole entity during multi-object editing, without considering that editing pairs may have varying levels of editing complexity and therefore require different optimal inversion steps. 2. The new search metric introduced in this article is simple yet effective. The approach of adding the two evaluation indicators makes sense, and visual verification confirms that the search metric aligns with the editing effect. In single-object editing tasks, employing the search metric yields promising results, comparable to other image editing methods. 3. The OIR introduced in this article is novel and effective, presenting a new solution for the multi-object editing task. Unlike previous approaches, which treated the entire image as a whole, OIR breaks down the task into editing pairs. The experimental comparisons with other methods are extensive and thorough. 4. The paper's structure is well-organized and easy to follow. The figures are well-designed, effectively illustrating the ideas and claims presented.

Weaknesses

1. In [a], it is mentioned that the inversion step can be considered as a fixed hyperparameter. However, the author only presents the results of the optimal inversion steps for example images, without demonstrating the overall distribution trend of the optimal inversion step across the entire dataset. Moreover, what are the distinct characteristics of editing pairs with larger and smaller optimal inversion steps respectively? 2. In the reassembly strategy of OIR discussed in this paper, both reassembly step and re-inversion are mentioned. The paper indicates that both methods can smooth the edges of images and enable global information interaction. Re-inversion first inverts the spliced latent and then denoises it, essentially increasing the denoise step. Is it possible to replace the operation of re-inversion by increasing the reassembly step? [a] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, ICLR 2022.

Questions

see weakness

Rating

8: accept, good paper

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Summary

The paper proposes object-aware inversion and reassembly for image editing. The motivation is that the inversion steps vary from the editing of different objects. Therefore, we need to choose different inversion steps for different objects. Also, for different objects in one image, we need to merge the editing results. Then the reassembly strategy is introduced. The proposed method achieves state-of-the-art performance.

Strengths

1) The paper is well-organized and easy to follow. 2) It makes sense to use different diffusion steps for different objects when performing the image editing. 3) The paper proposes a reassembly strategy to merge the editing results. 4) The proposed method achieves state-of-the-art performance.

Weaknesses

1) To determine the optimal inversion steps for image editing, we need to inverse the image to all steps and then edit them accordingly. It is time-consuming and not automatic. 2) The two contributions are more like the engineering stuff. However, admittedly, they do bring a lot of performance gain.

Questions

Please see my concerns in the weakness part.

Rating

5: marginally below the acceptance threshold

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Major Response

__Q1: The method is time-consuming (Reviewers xVir, r1AF and nVsr).__ A1: | Method | Ours (2 GPUs) | Ours (1 GPU) | Null-text Inversion | Plug-and-Play | Stable Diffusion Inpainting | DiffEdit | | --- | --- | --- | --- | --- | --- | --- | | Average time cost | 155s | 298s | 173s | 224s | 6s | 15s | | Maximum GPU usage | 19.75 GB | 19.75 GB | 22.06 GB | 7.31 GB | 10.76 GB | 13.51 GB | - We have introduced a parallel acceleration solution for our method in Appendix A.4 of the initial submission, which is recognized by Reviewer nVsr. In the new version of the paper, we move the details of the parallel acceleration method into "Acceleration for generating candidate images" in Section 3.1, and these details are highlighted in blue. - We observe that the sequential optimal inversion step searching process in OIR is time-consuming. To address this issue, we propose a splicing strategy over the denoising process. Firstly, we pair different denoising processes with varying inversion steps to achieve equal total steps. For example, the denoising processes with 1 and 50 inversion steps are paired. Similarly, the 2 and 49-step inversions are paired. In this way, denoising processes of the same total length can proceed simultaneously for parallel acceleration, as there is no dependency between denoising processes. In addition, the optimal inversion step search of different editing pairs can be parallelized on multiple GPUs to achieve further acceleration. - The table above presents a comparison results of editing speeds. It reveals that our method can be significantly accelerated in multi-GPU scenarios, whereas other methods lack clear parallel acceleration solutions due to the temporal dependency between the denoising steps. Detailed content is presented in blue font in Appendix A.6. __Q2: The contribution and significance of the method is limited (Reviewer r1AF).__ A2: We identified and addressed a fundamental challenge in multi-object image editing tasks, where existing methods typically apply the same inversion steps for all editing pairs, resulting in suboptimal editing quality. To this end, we propose a new editing paradigm to enable object-aware fine-grained editing, which integrates two tightly coupled components: - A simple yet effective new search metric that automatically and dynamically selects the optimal inversion step for each editing pair, unlike previous inversion-based methods that set the inversion step as a fixed value. This approach effectively balances the trade-off between editability and fidelity. Moreover, we can easily accelerate this process by splicing denoising processes of varying lengths into equal lengths and running denoising processes of the same length in parallel. - A novel disassembly and reassembly paradigm that is specially designed to disentangle the denoising process for each editing pair, which is not featured in prior works. The significance and contributions of our paper have been well recognized by Reviewer pgMn and Reviewer nVsr.

General response

We sincerely thank all reviewers for their valuable comments. ## Novelty Reviewer pgMn, Reviewer nVsr and Reviewer r1AF well recognize the novelty of our method: - "*This paper identifies a fundamental challenge in multi-object image editing tasks. The new search metric introduced in this article is simple yet effective. The OIR introduced in this article is novel and effective, presenting a new solution for the multi-object editing task.*" (Reviewer pgMn) - "*This paper identifies an interesting and useful phenomenon that the structure of different objects corresponds to a different level of latent as modeled in the diffusion process. Since the denoising process of different image parts is disentangled, the proposed method supports fine-grained multiple object editing, which is not featured in prior works.*" (Reviewer nVsr) - "*The idea of working on each object separately and reassemble is a interesting way to do multi-object editing.*" (Reviewer r1AF) ## Promising Results All reviewers agree that: - "*The proposed method achieves state-of-the-art performance. They do bring a lot of performance gain.*" (Reviewer xVir) - "*Employing the search metric yields promising results, comparable to other image editing methods. The experimental comparisons with other methods are extensive and thorough.*" (Reviewer pgMn) - "*Results are competitive compared to other inversion based editing techniques.*" (Reviewer r1AF) - "*The image editing results are indeed impressive, as shown in the main text and the appendix. Both the quantitative and qualitative results demonstrate the advantage over strong prior works.*" (Reviewer nVsr) ## Summary of changes We have revised our submission and summarized our updates as follows: 1. We have provided image editing speeds for different methods in Appendix A.6. (Reviewer xVir, r1AF and nVsr) 2. We have presented the image editing results of our method under different mask generators in Appendix A.7. (Reviewer r1AF) 3. We have supplemented the comparison results between our method and the inversion-based method with a segment model in Appendix A.10. (Reviewer r1AF) 4. We have provided the experiment to visualize the distribution of optimal inversion steps in Appendix A.9. (Reviewer pgMn) 5. We have presented the combined approach and the experimental results of our method with the inversion-based image editing method in Appendix A.8. (Reviewer nVsr) 6. We have revised the paper to enhance clarity in expression. (Reviewer nVsr)