OBJECT 3DIT: Language-guided 3D-aware Image Editing

Summary

The paper studies the problem of object-centric image editing. The authors first curate a dataset based on Objaverse by selecting high-quality textured samples, and then simulate+render them on a plane. The objects can be manipulated in 3D and rendered correspondingly, which generates the groundtruth for training learning-based object editing models. The paper further presents a diffusion-based object editing model (3DIT) based on zero-123, where the major difference is the addition of editing prompt conditioning. Results in quantitative and qualitative experiments show 3DIT outperforms baselines based on foundation models.

Strengths

The dataset curation using 3D simulation for the object-centric image editing task makes sense. This task intrinsically requires understanding of the 3D world and 3D-aware image formation process. The proposed dataset is guaranteed to be 3D-correct, and would be useful for research along this direction. The proposed method also achieved great performance compared to the baselines. Besides, the paper is well-written and easy to follow.

Weaknesses

1. The realism of the generated dataset is still limited: a) The single directional light (why not multiple lights or Image-based lighting?) which makes the shadow and shading's distribution not diverse/realistic enough. b) The size range of the object is quite limited (0.8 ratio threshold), in practice, there are a lot of cases where different sized objects are placed nearby (also the true size seems not kept, so chairs and lamps appears to be of similar scale). The authors indeed show some qualitative results of direct transfer to real data, but no 1) quantitative evaluation of sim2real or 2) comparison to baselines' generalization have been provided. 2. Although the data is generated with 3D, the method follows a 2D design. Such a model does not exhibit a great understanding of 3D objectness -- for example, the identity of the object is sometimes not well-kept during rotation or translation (e.g. the change of coke/headphone texture in Fig 1), or the cast shadow or the shading do not make full sense w.r.t. the whole scene if we look close enough (e.g. first 3 rows of clevr in Fig 1). Having some systematic analysis of the failure cases here would have been very helpful for future research. Minor questions/comments: 1. Why is the background always black? This seems to be creating domain gap, as usually we don't have pure black background in real life. 2. Wrong number highlight in Table 1, 3DIT(Multitask) - insertion - LPIP, 0.585 is worse than the other two. 3. The multi-task model is much worse in terms of FID (Table 1), but not other metrics. Are there any specific reason for this?

Questions

Please see my questions in the weaknesses. Overall, I think the pros of this paper outweigh the cons -- the curated dataset would be useful for future exploration along this direction and the proposed method can serve as a strong 2D baseline. I believe having sim2real comparison and systematic failure case analysis will further solidify the paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

More analysis on failure cases and limitations will be beneficial.

Summary

This paper constructs a dataset containing 400K examples, which is used for the task of language-guided 3D-aware image editing. This paper also proposes a model, named 3DIT, to solve this task. The model is based on 2D diffusion model, which first goes through the pre-training of text-to-image generation and Zero-1-to-3, and then is fine-tuned on the dataset for the 3D-aware image editing task. The model is evaluated on the proposed dataset and achieves state-of-the-art performance.

Strengths

1. This paper constructs a large-scale dataset, which is a good contribution. And language-guided 3D-aware image editing is an important task. 2. The paper is well written and easy to understand. 3. The experiments show that the proposed model trained on the proposed dataset have a reasonable ability on the task of language-guided 3D-aware image editing.

Weaknesses

My biggest concern is that the performance of the proposed model is not very good. 1. The GIF results in the supplementary material exhibit incorrect results on the shadow. 2. The lighting and shadow of experimental results on real-data is not realistic. The quality of edited images degrade a lot. Due to the poor performance, I am not sure if the quality and diversity of the proposed dataset is good enough. The illumination and background of presented real data is too simple. It would be great to add more challenging test data to see the upper bound of the proposed model.

Questions

Please see the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed the limitations.

Summary

The paper formulates a task of 3D aware editing using the language guidance. The task aims to insert, remove, translate or rotate objects in a scene (2D images) by maintaining the details like shadows, 3D consistency of the object, changes in the object sizes due to perspective projections etc. The model is based on Stable diffusion, Zero 1-to-3 method and fine tuning on the given dataset having editing information and text to describe the edit. The paper promises to release the dataset OBJECT derived from Objaverse, which the authors use to train their model on. The results in the teaser figure and others show that the model is able to perform the given edits, while preserving the semantics of the image. For examples, the objects are translated and it respects the perspective projection, shadows and placement on the surface. The authors also claim that the method is generalizable to the real images.

Strengths

1) The paper is able to show that the manipulations possible with the method can preserve the 3D properties of the scene including localization of the objects, scaling, shadows and consistency of the inpainted regions. 2) The paper compares with the image based baselines, for examples uses SAM to segment the images and translates the objects in the scene. It also compares with the a 3D baseline for rotation using Zero 1-to-3. 3) The paper shows quantitative and qualitative results of their method comparing scores such as PSNR, SSIM, LIPIPS and FID between the original and edited images. The authors also conduct a user study to access the quality of lighting and geometry.

Weaknesses

1) The method to train on the given dataset is not clear to me. There is no pipeline figure to explain the stages of the training. The first two steps are previous works. The contribution which is in the third step is not explained properly in the paper. Is this fine-tuning stage similar to Zero-1-to-3? How was the editing sequence fed to the network? How is it 3D aware besides the Zero-1-to-3 training? Is the method's 3D consistency (for example in rotation) upper-bounded by Zero-1-to-3? 2) While the images shown in the paper show that some properties are preserved as the editing operations are done, the results in the gifs show some obvious flickering artifacts which do not respect the properties like shadows. This does not go well with the objective of the paper. Besides the problems with Stable Diffusion , where do the problems arise? 3) How is the quality drop if number of sequential operations are done? For example once can perform insertion-> rotation ->translation etc for the same/ different objects in the scene. How does the quality drop compare with the Stable diffusion image editing methods? This is more interesting to me. A 3D aware editing framework should be able to handle multiple sequential edits with consistent results. 4) Another baseline would to use a monocular depth estimation model (eg Zoedepth, Midas) to extract the surface and perform the edits using Zero-1-to-3. This can handle the perspective projection of the objects, and/or even lighting and shadows. Did the authors try similar more strong baselines? How do they compare with the current method? 5) The paper claims to generalize to the real scenes. This is a significant claim and needs to be evaluated. The issue of synthetic and real image domain gap is an active research area. How does the current method solve that in this particular task? Were real images considered in Table 2?

Questions

Please refer to the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors do not discuss the limitations of their work in detail. Please add a detailed section on where the method fails.

Summary

The authors propose a large dataset of 3D aware image edits along with editing instructions built on the objaverse dataset. They also introduce a model finetuned on Zero-1-to-3 for 3D aware editing tasks which include object insertion, removal, translation and rotation. Comparisons are provided against state of the art models for each task and performance improvement is demonstrated.

Strengths

1. **Clarity**: The paper is well written with attention to detail. All the necessary details particularly with regards to the dataset creation have been adequately explained. 2. **Interesting dataset**: The 400k dataset of images along with edit instructions would serve as an interesting training and benchmarking dataset for the task of 3D aware editing. 3. **Quantitative metrics**: A number of qualitative comparisons and user studies are provided to demonstrate the geometric consistency of the edits and lighting consistency.

Weaknesses

1. **Novelty**: Although the proposed dataset represents an important contribution, the proposed approach relies on zero-1-to-3 and finetuning on a new dataset. 2. **Need for zero-1-to-3**: The approach finetunes a model on top of zero-1-to-3 to incorporate edit instructions. Can the finetuning be done on top of base SD?. Adding an ablation to this effect will be helpful to demonstrate the need for the 3 stage curriculum. 3. **Related work**: Several related work that may provide important context are missing. The authors might find some of the following works relevant and interesting [1,2,3,4]. Although some of these works are pre-prints and do not warrant strict comparisons, incorporating them into the related work section would place the proposed work appropriately w.r.t the landscape of current literature 4. **Changes in the edited image** : There are certain global changes in the edited image, that dilutes some of the claims w.r.t editing. Particularly, for the CLEVR dataset, the provided supplm examples show changes in the color of certain objects upon insertion/ removal [1] ControlNet [2] InstructPix2Pix [3] InstantBooth [4] GLIGEN

Questions

1. For translation and insertion, what are the location? are they provided in pixel space? Since they are 3D aware, does it make more sense to provide edit instruction in camera coordinates? ( In pixel coordinates, how is depth interpreted?) 2. Is it easier to provide control inputs as a spatial map (say as gaussians/ keypoints on a 2D image) rather than as a pixel locations to allow for more local specificaition?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Adequate treatment of limitations have been provided.

Summary

The 3DIT model is a language-guided 3D-aware image editing tool that allows for effective object editing while considering scale, viewpoint, lighting, and object occlusions. The model builds upon previous work in scene rearrangement and image generation, and incorporates a diffusion process to render object transformations. The authors conducted human preference evaluations to measure geometric and lighting consistency, and found that 3DIT outperformed relevant baselines in both categories. One of the key strengths of 3DIT is its ability to add, remove, or edit shadows to maintain consistency with scene lighting. This is achieved through a shadow generation module that takes into account the position and orientation of the light source, as well as the geometry of the objects in the scene. Additionally, 3DIT accounts for object occlusions by using a novel occlusion-aware rendering module that predicts the visibility of each object in the scene. The authors also introduced a new benchmark dataset called OBJECT, which consists of 3D scenes with multiple objects and associated natural language descriptions. They trained 3DIT on this dataset and found that it generalized well to images in the CLEVR dataset as well as the real world. This demonstrates the robustness and versatility of the model, and suggests that it could be applied to a wide range of real-world scenarios. Overall, by enabling users to edit objects in a natural and intuitive way, 3DIT opens up new possibilities for creative expression and visual communication.

Strengths

- The 3DIT model is a novel approach to language-guided 3D-aware image editing that builds upon previous work in scene rearrangement and image generation. The model incorporates a diffusion process to render object transformations and uses a novel shadow generation module and occlusion-aware rendering module to maintain consistency with scene lighting and object occlusions. - The authors conducted human preference evaluations to measure geometric and lighting consistency, and found that 3DIT outperformed relevant baselines in both categories. This demonstrates the effectiveness of the model in producing high-quality, visually consistent image edits. - The authors trained 3DIT on a new benchmark dataset called OBJECT and found that it generalized well to images in the CLEVR dataset as well as the real world. This suggests that the model is robust and versatile, and could be applied to a wide range of real-world scenarios. - The potential applications for this technology are vast, including virtual and augmented reality, gaming, and e-commerce. The model could be used to create personalized avatars for virtual reality environments, or to generate realistic product images for e-commerce websites. Additionally, the model could be extended to support more complex scenes and interactions, such as object physics and collision detection.

Weaknesses

- The ablation study is relatively weak. It's unknown which component contributes most to the final performance and which is effective. - The shadow of the box did not follow the rotation action in the shown GIF. And some artifacts are obvious. - How did the method choose which box to be moved? Or did it need a handcrafted mask as the selection? - What is the number of samples used for calculating FID? Normally FID is not reliable of the number of samples is small. - The comparing table did not include the previous method. I believe several important baselines [a] methods are missing. References: [a] Editable free-viewpoint video using a layered neural representation

Questions

N/A

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

See above

Response to rebuttal

The authors do a great job of addressing most of the pressing concerns. I have additionally gone through the other reviews and agree that the provided dataset has some merits. However, I am still not entirely convinced about the technical novelty of the approach. Although performance trends w.r.t to SD initialization vs Zero-123 initialization are insightful and help highlight the need for the additional viewpoint based finetuning step, it is unclear if the performance difference for these task come from the viewpoint awareness injected by zero-123 or from more data that it sees during finetuning. The edits themselves although not multiview consistent provide a method to generate useful paired data. Additionally, as mentioned above the proposed dataset (if released) would make for a good benchmark. To that end, I would keep the current score and encourages the authors to release the dataset.