3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D Detection

Summary

This paper proposes a data augmentation approach to assisting the training of monocular 3D detectors by inserting 3D objects into indoor scenes in a physically plausible manner. Specifically, it addresses two main challenges in the entire pipeline: 1) where and how to insert those objects; 2) which illumination on the object makes the rendering photorealistic. Experiments validate that the data generated by the overall pipeline can enhance the state-of-the-art monocular 3D detectors. Detailed ablation studies further provide some insights regarding which aspects are important in the proposed method.

Strengths

- The basic idea is easy to follow and the illustration figures are clear. - The overall pipeline is proven effective for enhancing downstream 3D perception systems. - This pipeline is systematic and comprehensive, taking almost most of the aspects when inserting 3D objects into scenes and image data generation for training monocular detectors. - The key insights regarding three challenges in the introduction and two critical considerations (where and how & illumination) in the methodology are accurate. - The proposed method achieves new state-of-the-art on the SUN RGB-D benchmark and has detailed ablation studies to reveal which aspects are most essential in the overall pipeline. (For example, the analysis in Table 4 is interesting.)

Weaknesses

- (Related work) The related work section has many inaccurate statements, such as MV3D is a multi-view method incorporating both LiDAR-based point clouds and images and VoxelNet is a LiDAR-only method, which should not be discussed in the monocular 3D detection section. In contrast, there are many missing works of monocular 3D detection in driving scenarios, such as 3DOP[1], MLFusion[2], M3D-RPN[3], MonoDIS[4], Pseudo-LiDAR[5], FCOS3D[6], SMOKE[7], RTM3D[8], PGD[9], CaDDN[10], etc. There is also some missing literature for 3D Data Augmentation, such as MoCa[11], GeoAug[12], etc. - (Methodology) The overall pipeline is the main contribution of this paper. From another perspective, the most important part is to combine different "existing" techniques together and make it finally work to produce photorealistic images after inserting 3D objects. The key flaw here is that most of the techniques used in each stage are not newly proposed in this work, making this work more like an engineering one. (although I admit that the overall pipeline is still a good contribution to the community) - (Experiments) The proposed method is not limited to any method and any dataset, but only tested on ImVoxelNet and SUN RGB-D. Although it can demonstrate the basic effectiveness, it would be much more convincing if the author can provide more ablation results on other baselines and datasets. [1] 3D Object Proposals for Accurate Object Class Detection [2] Multi-Level Fusion Based 3D Object Detection from Monocular Images [3] M3D-RPN: Monocular 3D Region Proposal Network for Object Detection [4] Disentangling Monocular 3D Object Detection [5] Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving [6] FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection [7] SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation [8] RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving [9] Probabilistic and Geometric Depth: Detecting Objects in Perspective [10] Categorical Depth Distribution Network for Monocular 3D Object Detection [11] Exploring Data Augmentation for Multi-Modality 3D Object Detection [12] Exploring Geometric Consistency for Monocular 3D Object Detection

Questions

None.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The author discusses the limitations and social impacts in the supplementary material.

Summary

This work introduces 3D Copy-Paste, a physically plausible indoor object insertion technique for automatically generating large-scale annotated 3D objects. This approach ensures the plausibility of the objects’ physical location, size, pose, and illumination within the scene Using this 3D copy-paste as augmentation, a better monocular 3D detector can be trained. Experiments are demonstrate on the SUN RGB-D dataset, demonstrating the effectiveness of the proposed object insertion method in improving the 3D detection.

Strengths

1. The method is clearly explained 2. The proposed methods for object insertion can help create annotated data for free. 3. The inserted object is physically plausible in location, size and illumination. Moreover, it's fully automated. 4. Solid experiments are conducted in SUN RGB-D based on a SOTA detector (ImVoxelNet), and the detection AP is improved using the proposed method as augmentation. (40.96-->43.79)

Weaknesses

This work may somewhat lack novelty. I'm not familiar with the indoor scene object insertion task, but for the outdoor scene, there are quite a lot similar works[1-4]. It would be better to include these works in the literature review. These works share similar ideas with the authors, where they insert object in physically plausible location and illumination. And some works also test with downstream tasks and demonstrate effectiveness of using actor insertion as data augmentation. Besides, I'm not fully convinced by the claim that indoor scenes are "more challenging" in L51. The layout, illumination, complexity of outdoor scenes are apparently more complex. [1] Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes [2] GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving [3] Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion [4] 3D Data Augmentation for Driving Scenes on Camera

Questions

I'm not familiar with the ImVoxelNet,I just quickly went through it to review this paper. ImVoxelNet reported AP@0.15. Why AP@0.25 is reported in this paper? I will raise my rating if the authors can apply this approach to outdoor scenes like kitti and nuscenes and improve ImVoxelNet results.(ImVoxelNet is already tested on this dataset)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Apparently, the authors did not go over https://neurips.cc/public/guides/PaperChecklist. Not many experiments and implemented details are provided, Licenses and Assets are not clearly described either. But I don't penalize it in the rating though.

Summary

This paper presents a novel 3D augmentation method to augment the variety of 3D scenes with the corresponding 2D images. This augmentation method focuses on addressing the data-hungry problem when doing monocular 3D detection. The strategy here is intuitive: to insert 3D synthetic object models into 3D real scenes to augment the 3D scene data, and make sure consistent illumination, shading, and layout reasonability without any object collision issues. Therefore, the pipeline in this paper consists of three parts to answer three questions: 1. where and how to place the object in a 3D scene; 2. what is the illumination and how to add it onto object; 3. use the augmented dataset for network training. In my view, the major contribution of this paper is the pipeline or concept: to leverage inverse rendering and re-rendering to augment 3D data and its corresponding 2D image for monocular 3D detection. Each module used in the pipeline already exists. The experiments in this paper are pretty extensive, and this paper is well written.

Strengths

As discussed above, the major strength of this paper is its concept and the tailored pipeline. 1. This paper proposed a new strategy to do 2D-3D data augmentation using neural inverse rendering and re-rendering. 2. The tailored pipeline successfully verified the idea, that such a data augmentation strategy can improve monocular 3D detection by a large margin.

Weaknesses

In my view, the weakness is from the method contribution side: 1. Neural inverse rendering to support object editing and augmenting is not novel. There are many works in image-based rendering and decomposition that can support inserting new objects into a 3D scene. I understand that the contribution here is to use it for data augmentation to support monocular 3D detection. But the methodology here is not novel. 2. Many modules are from existing works (e.g., the inverse rendering framework, plane extraction) which makes this paper more like a novel combination to improve an existing task. I like the insight here by leveraging inverse rendering to do data augmentation for monocular 3D detection, and the performance is quite good. It would be more convincing if it also works for other datasets, e.g., ARKitScenes. Because in my view, the experiment performance is the other major contribution.

Questions

1. I wonder how to choose the object class category for insertion. Is it randomly sampled from an object class set? or manually? Because for indoor scenes, there is a strong scene context, e.g., it is not much likely to insert a "bed" in a "toilet". How do you consider such consistency?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors discussed the limitation and societal impact in the supplemental.

Summary

This paper addresses the scarcity of large-scale annotated datasets which is challenging for rapid deployment of monocular 3D object detection models. A physically plausible indoor 3D object insertion approach is proposed to automatically copy and paste virtual objects into real scenes. The resulting objects have 3D bounding boxes with plausible physical locations and illumination, augmenting existing indoor scene datasets. The proposed data augmentation method achieves state-of-the-art performance in monocular 3D object detection. Most importantly, in this approach, a candidate selection process is applied along with a spatially varying illumination procedure from an existing method to ensure the plausibility of the objects’ physical location, size, pose, and illumination within the scene. From the results, the location and illumination of the inserted objects appear to have a significant impact on the performance of the downstream model.

Strengths

This paper's proposed approach is impressive. It automatically inserts virtual objects from ObjaVerse into real scenes, addressing the issue of limited annotated datasets in computer vision. The method ensures the objects' physical location, size, pose, and illumination, resulting in augmented indoor scene datasets. The use of plane selection methods, discarding objects with collisions using a simplified assumption, and constrained parameter search for insertion, along with the use of spatially-varying lighting estimation is well-thought-of and designed process. Its improved performance from resulting augmentations for monocular 3D object detection demonstrates the effectiveness of the method. Furthermore, the paper is very well-written and easy to follow.

Weaknesses

The paper lacks a comparison or discussion in relation to Common 3D Corruption (CVPR 2022; not cited). Even though common 3d corruption only evaluates 2D downstream tasks, it could still be utilized and compared for 3D object detection and demonstrate how important is 3D and illumination-aware physically grounded insertions. Another weakness is that the paper only evaluates one task. It would be beneficial to assess the impact of 3D grounded and illumination-aware object insertion on other 3D or 2D tasks. Could it also enhance 2D recognition tasks such as segmentation or detection? Including these aspects in the paper would reinforce its findings. It appears that the paper is lacking a basic 2D copy-paste baseline as well. It would be interesting to see how much this technique could potentially improve 3D tasks (Q: Is random insert in comparisons a 2D insertion or 3D insertion?). Additionally, the paper is missing ablation results for changes in object detection performance for each individual object.

Questions

I have a few questions that I hope the authors can help with. - First, I'm curious about what happens to out-of-distribution insertions such as the NYU dataset. Would it be helpful to include more datasets? - Second, I'm wondering how long it takes to render one object, taking into account the time it takes to search for the insertion point, correct the appearance, and other factors. - Lastly, I've noticed that most of the inserted objects are generally diffuse. I'm curious about how the object insertion process is affected when inserting shiny or specular objects.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Missing comparison with common 3D corruptions as data augmentation and comparing 2D recognition tasks + other 3D downstream tasks.

Summary

The manuscript introduces system to create RGBD datasets augmented with additional 3d objects (one per rgb frame). The inserted objects are placed such that they dont intersect other objects, stand on the ground floor and the objects are rendered into the original RGB frame such that the lighting of the scene is respected. The augmented dataset can then be used to train slightly better 3d detectors since more 3d object examples can be supervised.

Strengths

The manuscript tackles an important problem for 3D object detection research: the scarcity of supervision data and the difficulty inherent in augmenting in 3D. The 3 factors for physical plausability make sense to me and the implementation of them in the paper seems pretty solid judging by the example images from the generated augmented dataset. The claim that the full system of physically plausible augmentation is needed is well supported by ablations against other plausible choices (random placement, simpler light sources). The system seems sound although I am not an expert on the most recent ways of estimating illumination.

Weaknesses

The primary weakness to my mind is the use of only a single (small!) 3d object detection dataset with SUB RGB - ScanNet is substantially bigger and commonly used. Who knows maybe this method would be even more powerful on a harder dataset? The delta in mAP is not very big with the addition of the presented augmentation method (from 40.96 to 43.79). I question whether people will want to implement the full presented system in order for such modest gains. This limits the potential impact in my eyes and means claim (2) "significant" improvements is not fully supported. ImVoxelnet is not a single-view detector (it is multiview but can be used for single-view detection); In the original paper, the mAP for SUN RGB-D is actually 43.66 (I suppose thats because of the different IoU threshold of 0.15 but it raises the question to me why the evaluation in this manuscript doesnt use the same one?) I am not 100% clear on the way the illumination is estimated and used as somebody who has never worked with such methods. This would make it hard for me to replicate the presented work. There is a few sentences on semantic plausibility and that it didnt help but its not well explained how semantic plausibility was achieved. This is an important negative result that I am not sure I can trust at the current level of explanation.

Questions

see weaknesses

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations such as the need for metrically scaled objects, the need for depth images and the knowledge of the gravity direction are not explicitly discussed in the paper.

response

Thank you for addressing my questions and concerns. "To allow easy usage, we will release our code, model, and generated data." will be critical to enable others to benefit from this paper. Based on this promise and the clarification around the mAP improvement and the other reviewers responses I am happy to increase my rating to weak accept.

Response to Rebuttal

I acknowledge that I have read the authors' rebuttal and the other reviews. Thanks for your rebuttal and I feel most of my concerns are addressed. Given my impression that this paper has done solid work but may have relatively small technical contributions, I will keep my score in the final decision but support its acceptance considering its value to this community.

Response to author's rebuttal

Thank you for your comprehensive response addressing the concerns raised in my review. I appreciate the clarifications provided and the commitment to making the suggested changes in the revised version of the paper. I look forward to seeing the updated manuscript and the enhancements you plan to incorporate based on the feedback and other reviews.

Thanks for the response

Thank you for the response, especially on the new metric setting and results on ScanNet. After reading others reviews and response from author, I plan to maintain weak accept decision. Overall this work is decent without significant flaws. The reasons preventing me from raising rating are: 1. No experiments on outdoor scenes, which is the biggest consideration for me to raising score before rebuttal. 2. I hold similar ideas to 7hAU on that it may not be necessary to build such a complicated system, just to improve the mAP a bit, though I understand that it's hard to to achieve such improvements. One suggestion for the work is that to also take realism as consideration, and may add some non-paired metrics like FID and human evaluation. Best