Summary
This paper proposes a new method for multi-view consistent inpainting. The input includes a video and a sequence of masks. The first image could be manipulated by any 2D editing approach, while the remaining images need to be inpainted using the proposed method. The method is built upon the SD1.5-inpainting model and fine-tuned with two LoRAs: one for object-centric and one for scene-level, using different datasets. It employs motion priors by fine-tuning the domain-adapted LoRA and temporal transformers from AnimateDiff, pre-trained on a video dataset. The technique of reference key & value concatenation from previous works is used to enhance consistency with the reference image. Flow features are extracted and added to the UNet as additional conditions. During inference, a heuristic strategy is proposed to warp and propagate the mask from the first frame to all subsequent frames.
Strengths
1. The task of propagating a single 2D edited image to generate a consistent video is interesting.
2. The overall pipeline is well-motivated, built upon the SD1.5-inpainting base model, fine-tuning two LoRAs on different datasets, and including temporal modules, flow features, and reference attention techniques.
3. The proposed method doesn't require camera pose as input and has a short inference time.
4. The paper evaluates the method on several datasets with various tasks and also provides some ablation studies.
5. It's interesting to know that the dense feature is less effective than the slot embeddings.
Weaknesses
1. While the title contains "to Bridge 2D and 3D Editing," the proposed techniques are primarily designed for 2D images or videos, with no explicit 3D representation used or 3D output generated. It would be interesting to explore whether 3D output can be extracted from the resulting image sequence. Otherwise, it might be better to change the title "3D editing" (and other mentions in the text) to something less misleading.
2. The utilization and fine-tuning of domain-adapted LoRA and temporal transformers from video models are intuitive and straightforward, but there do not appear to be any insightful design choices, limiting the novelty.
3. The technique of "Reference Key & Value Concatenation" has been used in many previous works. We fail to see much novelty here.
4. While the section introducing the techniques is generally clear, the concrete settings and details for training, inference, and evaluation are very hard to follow. I tried my best to understand the setup but still feel confused about many details, which is quite frustrating. The paper needs to be rewritten in a more well-organized way, introducing all necessary settings and details. Specifically, the following points need clarification:
a) During inference, you employ 2D models to generate reference images for different tasks (e.g., object removal/editing/replacement). However, what is the input-output pair during training? For example, do you have specific ad-hoc training pairs for the object removal task?
b) The experiment contains various setups, and the concrete settings are confusing. For each setup, it would be better to explicitly mention the task (e.g., inpainting with object/random masks, object generation, object removal, object replacement), dataset, the input, ground truth output, and how the **mask** are generated. Currently, it only says "Object-Centric Results," "Forward-Facing Results," and "Real-World 3D Scene Editing." But the concrete tasks are unclear. For example, what is the difference between "Object Removal" and "Scene-Level Inpainting"? I assume "object removal" is also achieved by inpainting? If the setup for quantitative and qualitative experiments is different, please also mention it. For example, there are some qualitative results for "object removal," but it should not be evaluated quantitatively (no ground truth)?
5. In equation (1), you concatenated both the masked latents and unmasked noised latents. Some related questions:
a) The 9-channel input differs from the original input of SD1.5-inpainting. How do you achieve that?
b) The noise-free latent is currently masked after VAE encoding. Will this cause leakage? It might be better to mask before VAE encoding.
6. What is the difference between the "Masking strategy" (line 163) and "masking adaptation" (line 228)? Is one for training and one for inference?
7. The method extracts flows before masking. This may be acceptable for the tasks of removal and replacement, but will it cause leakage for the task of general NVS and inpainting, considering the flow model may have seen the ground truth image?
Questions
1. Have you considered directly fine-tuning a video inpainting model instead of using SD1.5-inpainting?
2. Line 186: The sentence "which always captures the unmasked reference latent without noise (Eq. 1), thus it is unnecessary to re-scale the latent before adding noise from another U-Net" is unclear to me. Could you please clarify?
3. Line 207: The sentence "except that the former should be normalized in the query dimension, while the latter is normalized in the key dimension" is unclear. Could you elaborate?
4. Line 212: The phrase "with comparable performance and fewer trainable weights" needs clarification. Which two setups are being compared?
5. Could you provide more details about the 3D attention mechanism?
6. Line 309: In the context of not having object-level tracking masks, how do you obtain the mask?