Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like. In this work, we present a new form of editing, termed imitative editing, to help users exercise their creativity more conveniently. Concretely, to edit an image region of interest, users are free to directly draw inspiration from some in-the-wild references (e.g., some relative pictures come across online), without having to cope with the fit between the reference and the source. Such a design requires the system to automatically figure out what to expect from the reference to perform the editing. For this purpose, we propose a generative training framework, dubbed MimicBrush, which randomly selects two frames from a video clip, masks some regions of one frame, and learns to recover the masked regions using the information from the other frame. That way, our model, developed from a diffusion prior, is able to capture the semantic correspondence between separate images in a self-supervised manner. We experimentally show the effectiveness of our method under various test cases as well as its superiority over existing alternatives. We also construct a benchmark to facilitate further research.
Paper
Similar papers
Peer review
Summary
The paper proposes a method to modify specific parts of the image with the content from the reference image and consistently with the original content. It combines the technique of dual diffusion to replace the key and value features, which is used in MasaCtrl, of the source image with the reference image. The method is trained in a self-supervised manner, in which some parts of the image are masked, and the model is trained to recover them.
Strengths
- The paper is well-written and easy to understand. - The end-to-end pipeline doesn’t need fine-tuning for each image. - The proposed method is friendly for users that only give the source image, reference image, and mask, and then the model automatically gets the correct content from the reference image and aligns for the original part that is specific by mask. - Experimental results are impressive with various categories and metrics.
Weaknesses
- The proposed method is not actually zero-shot as it requires training. It requires expensive resources for the training model. - The method presented in this paper lacks novelty. It uses the old technique (i.e., replacing the key, value feature from MasaCtrl [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, ICCV 2023]) and combines with an old training strategy that recovers the region that is specific by mask. - The source and reference images must be the same scale as the object, and the source image must contain only one or a salient object.
Questions
What if the source image is a complicated image that contains some objects and the desired modified object is not a salient object?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
The paper is not so novel; it reuses components from existing techniques. The proposed method is not actually zero-shot as it requires training.
Dear Reviewer, We kindly remind you about our submitted rebuttal. We understand that your time is valuable, and we greatly appreciate the effort you put into reviewing our work. If you have any questions or require further clarification on any points, please don't hesitate to reach out. We look forward to your feedback and hope for a favorable consideration.
Dear Reviewer q758, As the deadline for the discussion phase approaches, we wanted to kindly remind you to read our rebuttal. We highly value your feedback and are eager to incorporate your suggestions to improve our work. If there are any aspects that require further clarification or additional information, we would be more than happy to engage in continued discussion and provide any necessary details.
Dear Authors, Thank you for your detailed rebuttal. First, I would like to acknowledge that I agree with your definition of the zero-shot method. Second, while I appreciate the detailed explanation of your contributions and their novelty, I believe that the novelty is primarily focused on user experiments related to image editing tasks. From your presentation in the paper, it seems that you are proposing a novel architecture (Imitative U-Net). However, I still maintain that the overall method may not be entirely novel. Finally, the examples provided in Fig. R1 do not fully convince me. Could you test a more complex example that includes editing smaller objects within the image? For instance, an image with a large number of overlapping objects or objects that do not interact with each other would be more compelling. In conclusion, I would like to keep my rating as 'BorderLine Reject.’
Thank you for your response. I believe that the contribution and novelty are still misunderstood. `` it seems that you are proposing a novel architecture (Imitative U-Net)`` We claim multiple times that **the dual U-Net design is not our novelty/contribution, instead, it is a common practice**. We do not claim contribution. As mentioned in line 121, this structure is widely used in many tasks that require a reference image [56, 17, 46, 6, 58, 45]. These works verified the advantages of "dual U-Net" over previous works like IP-adapter or controlnet. We leverage this type of design solely as our feature injection module and do not claim it as our novelty. `` However, I still maintain that the overall method may not be entirely novel. `` We are the **only work** that could realize local region editing with a reference image ***even without providing the reference mask***, for this motivation, please refer to "B. Motivation of getting rid of the reference mask" in the global rebuttal. In addition, we are the **only work** that trains the model using one frame to complete another frame to learn the correspondence. As **the whole task and training strategy** are completely novel, we believe the novelty is enough. Again, we would like to reaffirm our core novelty below. **Novel form for image editing.** Unlike the commonly studied task of "object insertion", our proposed "imitative editing" offers a new editing form, which does not require users to specify the region of interest from the reference image. Such a design sufficiently simplifies the editing process, because users only need to brush once (i.e., the source image) instead of twice (i.e., both the source image and the reference image), and hence clearly distinguishes our approach from existing methods. It is also noteworthy that getting rid of the reference mask is essential for part-level editing, making our algorithm more flexible in practice. **Novel training strategy.** To accomplish "imitative editing", it requires the model to automatically analyze the semantic correspondence between the to-edit region of the source image and the "full" reference image. For such a purpose, our approach benefits more from the training strategy but not the data. Concretely, we propose to leverage the temporal consistency and variation across video frames, and urge the model to recover the masked region of one video frame by only using the information from another frame without specifying the corresponding region. To our knowledge, our proposed training pipeline offers a brand new way of using video data for diffusion-based editing. **Benchmark for the novel task**. We construct a diversified benchmark that contains multiple tracks and covers a large variety of scenarios. Besides, we come up with various metrics to thoroughly evaluate the compare different methods. Our benchmark could be beneficial for further explorations for imitative editing. ``Could you test a more complex example that includes editing smaller objects within the image?`` Please notice that, in the second row of Fig.R1 of the rebuttal PDF, we have added examples for editing "the candle", "the hat", and even "the socks of a full-body human". **We have already shown these small things that occupy less than 5% pixels of the source images**. We kindly remind you that, as the "to-edit" region could be specified by users with a "source mask". In this way, **there would not be any challenges** in distinguishing the "to-edit" region even if the environment is complicated with distracters or this region is small. According to the rebuttal policy, the PDF files could not be updated. We would contact ACs to see if we could give you more examples. **Please check if this response addresses your concerns, if you have further questions, we are happy to discuss them.**
Dear Reviewer, We have added examples for even smaller objects and given the link to ACs, you may receive this link from ACs later. Here we describe the examples: - A camera shot filled with large numbers of fruits, where some fruits are partially obscured by others. We edited one of the fruits that was partially obscured. - A complex rural scene with more than ten people and ten dogs or sheep, where there is significant occlusion between the people and animals. We replaced one of the heavily obscured dogs with a tiger. - A windowsill filled with plants (more than 10), with some pots partially obscuring others. We edited one of the pots according to the provided reference image. Please note that editing very small areas is only a corner case with limited practical usage in real-world applications and is unrelated to the main focus of our paper. However, we have still provided the requested samples. We hope these examples fully address your concerns. If you have any new questions, feel free to reach out to us at any time.
Dear Authors, Thank you for your further explanations. My concerns are mostly addressed, but it is still not so convincing. I can only increase my rating to Borderline Accept. I already know that the dual U-Net design is not your novelty/contribution. One thing I want to point out is that the way your contribution is presented in the paper confused me. I don't think your method is actually novel; it is only novel in the context of user experiments. p/s: I haven't received your examples of small objects.
Dear Reviewer, Thank you for your acknowledgment. ``the way your contribution is presented in the paper confused me`` Thank you for pointing it out. To make our contribution clearer, we would add a small paragraph at the end of the introduction section to summarize our contributions and novelties. In addition, we would add an explanation paragraph in Section 3.2 to claim the dual U-Net is not the contribution of this paper in the revised version. `` It is only novel in the context of user experiments.`` First, image editing is an application-oriented task, and we formulate a more convenient editing form without the reference mask. We think "this novelty in user experience" is already quite important, and could make great contributions for the image editing community. Besides, this "novelty for users" could not be simply archived by applying existing techniques, it also requires the support of our novel training strategies. Our video-based training strategy is specifically designed for learning semantic correspondence without reference masks, which is important for realizing our "imitative editing". ``p/s: I haven't received your examples of small objects.`` We have already sent the link to ACs, maybe they have not read the message. As the discussion period is going to end we will not be able to make further responses. Please remember to ask ACs for the results if they forgot to send you the link. By the way, we believe that the examples in the second row of the rebuttal PDF like "candles" and "socks" are already small. Considering that the model receives a "source mask" to indicate the to-edit region, the difficulties for the model would be similar even for more complicated scenarios. In addition, we appreciate your constructive discussions and we would make our paper clearer according to your suggestions.
Summary
To achieve more precise imitative editing, the paper proposed a training framework, called MimicBrush. It randomly selects two frames from a video clip, masks some regions of one frame, and learn to recover the masked regions using the information from the other frame. Also, it constructs a benchmark, consisting two main tasks, i.e., part composition and texture transfer, to evaluate the performance comprehensively.
Strengths
- The setting for part-level image editing is a novel idea. I believe it will be very helpful for designers to leverage imitative editing in their use cases. - The paper provides comprehensive experiments to prove its effectiveness. - The paper is well-written and easy-to-follow.
Weaknesses
The definition of inter vs. inner is unclear.
Questions
1. How is rank calculated in table 2? 2. Regarding line #195, does it mean that when the randomly selected grid number is 4, it would drop 4x75%=3 for SIFT-matched features and 4x50%=2 for other regions? So, in total, it would drop 5 grids? 3. What does the gid ratio mean in line #261? 4. The paper structure can be improved. Sec. 4.4 and Sec. 4.2 contain the “Qualitative comparison”. I suggest having only one Qualitative comparison section and separating it into subsections: “comparing with others” and “diverse applications without comparing with others”. 5. The mask representation is hard to identify, e.g., the right case in the first row or the left case in the third row in Fig. 7. I suggest to change a clearer representation.
Rating
7
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have shared the limitations and potential societal impact.
Summary
The paper introduces a novel approach called imitative editing aimed at enhancing user creativity in image editing tasks. Traditional image editing often involves matching references to the source image, which can be challenging. In contrast, imitative editing allows users to directly draw inspiration from in-the-wild references without needing to precisely align them with the source image. This paper introduced a new approach called MimicBrush, a generative training framework that leverages video frame pairs to learn semantic correspondence and recover masked regions in images.
Strengths
--This paper proposed a novel task, Imitative editing, which is a new editing scenario that enables users to leverage diverse reference images without the need for exact matches, promoting more intuitive and less restrictive editing interactions. --The training strategy is self-supervised by utilizing two frames coming from the same video is intuitive. To ensure effective training, the paper further introduced data selection that select pairs without too big or too small variations, data augmentation to increase the task difficulty, and masking strategy to mask patches similar to reference in order to force the model to take guidance from reference for reconstruction. --This paper proposed a test benchmark containing part composition and texture transfer to evaluate the performance of imitative editing, which is good.
Weaknesses
--Regarding part composition and texture transfer, if I understand correctly, the proposed framework appears capable of simultaneously addressing both tasks. However, based on the training procedure, I am curious about how the model distinguishes between referencing parts and textures. This differentiation does not seem explicitly addressed in the training process, potentially leading to the model misinterpreting user intentions during the inference stage. --Regarding ablation experiment, this paper present the training strategies about training data, augmentation and masking method. It would be better to also ablate the necessity of image training data. And for all the ablations, it would better to have qualitative comparisons to illustrate how such training strategy affect the visual quality.
Questions
please see above weaknesses. Overall I think this paper is trying to address a new and interesting editing scenario, which will make image editing more convenient.
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have adequately addressed the limitations.
Dear Reviewer, We kindly remind you about our submitted rebuttal. We understand that your time is valuable, and we greatly appreciate the effort you put into reviewing our work. If you have any questions or require further clarification on any points, please don't hesitate to reach out. We look forward to your feedback and hope for a favorable consideration.
Thanks for the detailed response. The rebuttal addressed my concerns.
Summary
The paper proposes a new form of image editing, termed imitative editing. In this scenario, the user can edit the local area of the source image with a similar area to the reference image. This requires the system to automatically figure out what to expect from the reference to perform the editing. To achieve this goal, the author first designed a new generative network, dubbed MimicBrush. Then, the paper also constructs the benchmark from the video clip. The paper demonstrates both qualitative and quantitative results to validate the effectiveness.
Strengths
1. The proposed imitative editing is reasonable, novel, and inspiring. As explained in the paper, such an editing form can facilitate other real-world applications in fashion and product design. 2. The paper contributes a valuable benchmark for the proposed task, including diverse topics and examples, which can benefit later research. 3. The demonstrated examples show desired and good qualitative and quantitative editing results.
Weaknesses
# Motivation & Novelty **Unclear motivation and advantages**: The core learning algorithm of the paper is an imitative U-Net, a reference U-Net, and a depth model. However, the advantages and motivations of such designs are not clearly discussed and validated. For example, some other networks such as Controlnet and IP-adapter, can provide conditional information from the reference image. Why is injecting features from the imitative U-Net better on the proposed imitative editing task? **Unclear novelties in terms of learning compared with existing methods**: The proposed imitative task is new in that the system can edit part of the object whereas the latest methods mainly insert full objects. However, the novelty and difference seem to only lie in the training data (full object v.s. part object). The learning paradigm seems to be the same. Can these existing works do well in the proposed imitative editing task if trained with the collected part-level data? The essential difference **in terms of learning and formulation** between part-level editing and full-object editing is not addressed. My understanding is that they are essentially the same learning paradigm but in different concrete forms. # Formulation & Writing **Unclear notations**: 1. The paper proposes a new task and a new framework. However, the problem and algorithm are not rigorously formulated. For example, what are the input, output, and masks **mathematically**? 2. What is the training objective (loss function) in the pipeline? What is the loss used in the depth model? 3. For concatenation and injection of K, V from the reference U-Net, which layers are injected? Are cross-attention and self-attention processed in the same way? # Experiments & Evidence **Limited editing types**: The demonstrated editing types are limited. 1. For example, for the same source image, can the user edit different parts of the object with different reference images? 2. Does the proposed method support general image editing such as changing objects (cat to dog), or shape (round cake to square cake)? **Unclear motivation on the SAM data**: The paper mentions using the SAM data in training except for the Video clips. How does the SAM data help the learning in this task? I think after the strong augmentation, the spatial position of different objects does not change. So, how does this help to learn the correspondence between the source image and reference image?
Questions
Please refer to the weakness part. My main concerns are about the depth of discussion and motivation of the proposed network and task, and the writing and formulation that are not serious enough to reach the criterion of Neurips.
Rating
4
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
The author already addresses the limitations of the work.
Dear Reviewer, We would like to gently remind you of our submitted rebuttal. We understand the demands on your time and sincerely appreciate the effort you invest in reviewing our work. If there are any questions or if you need further clarification on any points, please feel free to reach out. We look forward to your feedback and hope for a positive outcome. Thank you for your time and consideration.
Dear Reviewer iASj, As the deadline for the discussion phase approaches, we wanted to kindly remind you to read our rebuttal. We highly value your feedback and are eager to incorporate your suggestions to improve our work. If there are any aspects that require further clarification or additional information, we would be more than happy to engage in continued discussion and provide any necessary details.
Response to the rebuttal
Dear authors, I have read the rebuttal and other reviewer's comments. Thanks for the efforts in the experiments and clarification! My comments mainly proceed from the novelties and contributions in learning, i.e., theoretical understanding and derivation, OR the validated design of approaches and insights. Thus, the main components (Dual-Unet, KV replacement) are existing techniques (as also stated in RWq758). I acknowledged the authors' insights on "the Reference U-Net preserves details better, as it provides feature maps with higher resolution" in rebuttal. However, this argument is generally beneficial for all image tasks and does not sufficiently address how it facilitates the proposed task. So, the paper provides me with an impression of using existing (or sort of new) techniques and strategies with new data to build an application. Since this is not a theory paper, I think the novelties in the design of approaches can be limited though I acknowledged the interesting editing effects added in the rebuttal. Besides, considering the overall formulation of the current version of the paper, I choose to maintain my score.
Dear Reviewer, Thank you for your response. However, we **disagree** with your comments, and **it seems that our rebuttal is ignored**. This comment does not involve the new information that we claimed in the rebutall. ``I acknowledged the authors' insights on "the Reference U-Net preserves details better, as it provides feature maps with higher resolution" in rebuttal. However, this argument is generally beneficial for all image tasks and does not sufficiently address how it facilitates the proposed task.`` We have responded in the rebuttal that: "We have compared different strategies in Tab.3 and Fig.6. "CLIP" represents the strategy of IP-Adapter, and "DINOv2" represents the strategy of AnyDoor. We observed that the Reference U-Net preserves details better, as it provides feature maps with higher resolution." In this case, we conducted detailed ablation studies to explore the structure and gave both qualitative and quantitative analyses in the paper. Considering this structure is **not our contribution**, we believe this experiment is already sufficient, and the advantage of this structure is exactly verified "in the proposed task". However, our claims around this experiment seem to be ignored, so we disagree with this comment. `` So, the paper provides me with an impression of using existing (or sort of new) techniques and strategies with new data to build an application`` We have repeatedly claimed in the rebuttal that **the dual U-Net is not our contribution or novelty** and carefully explained **the major differences are beyond the data**. However, these explanations seem to be ignored, and the new comments still focus on **the novelty of the Reference U-Net and training data**. It seems that the misunderstanding still strongly exists. To make our contribution clearer, we would add a small paragraph at the end of the introduction section to summarize our contributions and novelties. In addition, we would add an explanation paragraph in Section 3.2 to claim the dual U-Net is not the contribution of this paper in the revised version. We appreciate your comments and please carefully read our responses and the initial rebuttal to see if these concerns have been addressed. We would be happy to get your suggestions to polish our paper. Thank you.
Thank the author for providing further information and contributing great work to the image editing community! I have read other reviewers' comments and the rebuttal information. I would like to remain my rating as 'Accept.’
Thank you very much for your recognition of our work. We greatly value the feedback you provided, and we will follow the discussion to make this paper clearer and better.
Decision
Accept (poster)