Localize, Understand, Collaborate: Semantic-Aware Dragging via Intention Reasoner

Summary

This paper proposes a novel insight to transform the “how to drag” issue into a two-step “what-then-how” by introducing an intention reasoner and a collaborative guidance sampling mechanism. They also raise the problem of image quality issues and design quality guidance to enhance performance. Experiments show their superiority in semantic-aware drag-based editing.

Strengths

1. The issue of ‘‘inherent ambiguity of semantic intention’’ seems crucial and interesting. And shifting “how to drag” issue into two-step “what-then-how” can improve the controlling effect of the dragging operation. 2. The paper is well-written and easy to follow.

Weaknesses

Major： 1. How were the single experimental results (like Figure 4) selected from the diverse results conforming to the intention? If they were manually chosen, could this lead to unfair comparisons? 2. Providing the inferred potential intentions, source, and target prompts corresponding to each generated example may help explain the reasons for superior performance. For instance, in the third row of Fig. 5 on the left side, considering that the hand and the dragging operation should not be related to the corresponding prompt, why do other methods struggle with handling the hand information, while the proposed method can manage it effectively? Minor: 1. Fig. 5 caption, misspell of DragonDif(f)usion.

Questions

I believe that, fundamentally, the user's dragging operation has an intuitive single expectation, but the ambiguity of the operation itself leads to an ill-posed one-to-many mapping. Moreover, this mapping is difficult to traverse and may not even include the user's actual expectation among the diverse versions. Therefore, is it possible to further narrow down the range of expected dragging results, or allow the user to explicitly indicate their needs through additional operations? For other detailed questions and suggestions please refer to the major weakness.

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This paper aims to address the limitation of current dragging-based image editing methods that understand the intentions of users. To this end, the proposed method leverages the reasoning ability of LLMs to infer possible intentions, which are used to provide (asymmetric) semantic guidance in editing. Furthermore, a collaborative guidance sampling method that jointly exploits semantic guidance, quality guidance, and editing guidance is proposed. Among these three types of guidance, quality guidance is provided by a self-trained discriminator with an aesthetic score and images generated by a baseline. The experimental results on the DragBench dataset indicate that the proposed method outperforms several existing methods.

Strengths

1. This paper is well-written and the ill-posedness of dragging-based image editing is indeed a problem that should not be ignored. 2. The proposed LLM-based reasoner and the collaborative guidance sampling strategy improve the baseline performance.

Weaknesses

1. Despite the performance gain, there are observable artifacts in the synthesized images as follows: * Page 7, Figure 4, the last row, windows are missing. * Page 8, Figure 5, 1st row, right column, the shape and the texture of the mailbox are changed. * Page 16, Figure 7, 2nd row, wheels are changed; the shape of the target in the second last row cannot be preserved either. 2. Utilizing LLMs to infer intentions is interesting. However, it requires further clarification: * How can we ensure that the predicted outputs of LLMs (even with the highest confidence) really coincide with the intentions of users? * To the best of my knowledge, the selected baselines do not involve semantic priors from LLMs. Hence, the comparison might be unfair. To fully demonstrate the advantages of the proposed method, text/instruction-based methods should be included for comparison. * Even with the provided visualization of the ablation study (Fig. 6), it is still unclear how the introduced components affect the results. Sometimes the full model generates artifacts that do not appear in the results of the variants of the proposed method (e.g., the round bottom of the spinning top in Fig. 6 is changed mistakenly by the full model).

Questions

Q1: In A.8 the limitations section, why the proposed method is training-free as it includes a trainable quality discriminator? Q2: How is the computational complexity of the proposed method? Q3: How do we select the optimal one from the $n$ sampled outputs from the LLM? Q4: Except for the GScore metric proposed by GoodDrag, has any other commonly-used quality metric (e.g., LPIPS) used for evaluation? Q5: What will the results look like if the predicted intentions and dragging movements are contradictory? I might change my score depending on the responses to the above questions.

Rating

5

Confidence

4

Soundness

2

Presentation

4

Contribution

2

Limitations

Yes, the authors have discussed the limitations and the potential negative societal impact of the proposed method in the appendix.

Summary

This paper presents a novel framework called LucidDrag for drag-based image editing. Compared to previous methods, LucidDrag first reasons the intention of the draging operation using LLM (GPT 3.5) and provides a semantic guidance for the following editing process. For the better image fidelity, the authors also design a GAN-based discriminator as the quality guidance, obtaining improved results. The authors perform sufficient comparison experiments and ablation studies to support their claims and verify the efficiency of their proposed components. And the authors promise to release the codes.

Strengths

1. The writing is well except for some minor issues. 2. The authors note the problem that the drag-based editing has the inherently ill-posed nature, which is because multiple editing results may correspond to the same input image and draging conditions. Based on this observation, the authors attempt to introduce additional prompts to provide semantic guidance for better editing results. 3. The comparisons and ablation studies are sufficient to support the claims.

Weaknesses

1. It's a good idea to introduce additional prompts to help the editing process. However, why using an Intention Reasoner to "guess" multiple intentions and then sampling between them? Maybe a better way is that the users provides their true intention and LLMs are just used to normalize the prompts? So, I'm skeptical about the utility of the Intention Reasoner. 2. Some confusions: - Line 162-163, how do you get reasonable intentions only taking the generated description of the object of interest 'O', the original image caption 'C', and drag points 'P' as input? Take Fig.2 as an example, 'O' is "The nose of a woman", 'C' is "A woman", 'P' is the draging points. There is no information about the original image, so how can LLMs output intentions like "A woman looking up"? Where does the "looking up" come from? - Line 173, $z^{gud}_T$ should be $z^{gen}_T$? or the $z^{gen}_T$ on the left in Fig.2 should be $z^{gud}_T$? Maybe the notations in Fig. 2 should be modified. - Fig. 4, LucidGrag needs additional generated prompts as inputs. You should also demonstrate these prompts to help us understand the editing process. BTW, due to sampling generated intentions based on the confidence probabilities (Line 169), the generated results of LucidGrag should be different. Maybe showing different results can also help us to understand.

Questions

Refer to the weakness.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes.

Summary

The paper introduces a novel framework for semantic-aware drag-based image editing. Specifically, to address the limitations in understanding semantic intentions and generating high-quality edited images, it utilizes an intention reasoner to deduce potential editing intentions and a collaborative guidance sampling mechanism that integrates semantic guidance and quality guidance. Experimental results validate the effectiveness of the proposed method in producing semantically coherent and diverse image editing outcomes.

Strengths

1.The paper is well-organized and easy to follow. 2.The proposed what-then-how" paradigm for drag-based editing is novel and sound. 3.The proposed collaborative guidance is also interesting and has shown to be effective. 4.The proposed method is shown to outperform the existing methods on various editing tasks.

Weaknesses

1.It seems that the used LVLM and LLM models are not finetuned, and I am wondering how different models perform on the task. Are the confidence probabilities reliable or meaningful without any fine-tuning? 2.The efficiency of different methods should be provided for comparison. 3.The limitations are recommended to be added to the main paper.

Questions

Please refer to the weakness part.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are discussed in the supplementary material.