Our world is full of identical objects (\emphe.g., cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired by this observation, we introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects. SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances.An inverse graphics pipeline is subsequently employed to jointly reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances. Our primary contributions involve utilizing object duplicates as a robust prior for single-image inverse graphics and proposing an in-plane rotation-robust Structure from Motion (SfM) formulation for joint 6-DoF object pose estimation. By leveraging multi-view cues from a single image, SfD generates more realistic and detailed 3D reconstructions, significantly outperforming existing single image reconstruction models and multi-view reconstruction approaches with a similar or greater number of observations.
Paper
References (74)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The proposed method in Structures from Duplicates leverages multiple appearance of an identical objects in a single image to reconstruct the geometry and material properties of this object. Each instance is assigned a virtual camera, such that the shared object representation is aligned in the same space. Therefore multi-view reconstruction can be applied reconstructing each instance in a separate view. First each object is identified using a segmentation method and cropped from the image. Initial camera poses are estimated using Structure from Motion (SfM) method COLMAP on the cropped images of each segmented object. To avoid the issues that come with extreme differences in the view points for SfM methods, images of instances are rotated and optimal pairs identified. From estimated view point and image pairs the surface geometry following NeuS, the object's decomposed BRDF and environmental lighting following InvRender are optimized following known inverse rendering approaches. Results on synthetic data show comparable quality with multi-view reconstruction approaches, while using duplicates of the same object in a single image and present better performance in the multi-view case, leveraging virtual views from duplicates. Additional results on real data are presented that validate the result on synthetic data.
Strengths
The authors present an interesting solution to a common setting in the real world. The given introduction and related works motivates this setting of multiple instances well, which is an interesting setting by itself, that has not been explored by prior work in a similar way. The method explanation is thorough on all major details. Building on well-known and established decomposed neural rendering architectures and ideas (NeuS, simplified BRDF) is acceptable because this representation is not the main focus of the paper and the method does not lack to my understanding. Presented results justify most claims made in the beginning and ablation studies validate design choices. Especially showing that explicitly modelling lighting, geometry and material properties has an impact on the reconstruction quality over multi-view reconstruction on a single object is an important insight of the presented work. One could argue, that this paper lacks novelty and just applies known concepts, such as instances from repeating object classes in the same images and building a prior from that. In my opinion neglecting a pre-trained prior is a strong advantage of the paper.
Weaknesses
A major weakness of the presented paper is the variety in the results. The synthetic and real dataset are both on the smaller side and pretty homogeneous, as elaborated in the following: - While the motivation is convincing and Fig.2 shows a good variety of examples for the presented setting, the presented results in the main paper lack some variety and complexity that can be seen in the presented real world examples like the screws and chairs. The main paper presents only simple, closed object geometries, such as bottles, cans, apples, or the toy plane in the synthetic data with a lack of thin structures and complex textures. On the other side the alarm clock and coffee maker in the supplementary of this work show that the reconstruction can lead to overly smooth geometry on complex structures. This is expected on the limited number of views, but not properly addressed in the main paper. - Not purely a weakness, but a suggestion to strengthen the claims I would ask the authors to add results on more complex real geometries and scenes. To my best understanding this can be a complex task to capture and reconstruct, but it would have a positive impact on the limited scope provided by cans, bottles and packaging. Ablation studies on the number of objects from 2 to 10 objects on synthetic data show results, in which a higher number of objects significantly improves the results on almost all metrics of the reconstructed object. I would argue that even a higher number could improve results further and with respect to the presented motivation such experiments are not unlikely in the real world. Therefore an upper bound for improvements could further strengthen the paper.
Questions
Partially addressed in the weakness sections: - Were ablation studies performed on all objects (e.g. 2-10 instances per object category), please clarify?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
I think the authors adequately addressed all major limitations, such as a required instance segmentation and the reliance on COLMAP poses. Additionally they addressed potential solutions. I would suggest to add a short paragraph on the scope of the paper with respect to the experiments, that I suggested in the weakness section.
Summary
This paper presents a pipeline for recovering object shape and surface materials from one single of multiple identical duplicate instances. The pipeline first extracts instance masks, registers camera poses using COLMAP (with in-plane rotation augmentation) and recovers shape, materials and environment lighting through inverse rendering.
Strengths
### S1 - A sound inverse rendering pipeline - The proposed pipeline is carefully designed, starting from instance segmentation from Mask2former, to a camera pose estimation step using COLMAP with in-plane rotation augmentation, and finally to inverse rendering of shape, BRDF material and SG env map similar to [65]. - Given that the instances are mostly identical, this pipeline in general works well. ### S2 - Good writing - The paper is well written, with clear descriptions of the problem setup and most technical details.
Weaknesses
### W1 - Quite a trivial task assuming *identical* instances - Fig. 1: "the world is full of identical objects" - it might be difficult to argue that different instances that appear in one scene are *identical*. This is in contrast to the claim in [66], which says "no two roses in the natural world are identical". - This paper focuses more on manufactured and synthetic objects, which appear more similar thanks to the consistent manufacturing process, but in reality, different objects often vary in both shape and appearance, due to wearing or natural variations, for instance, multiple apples in a basket. - This is the fundamental assumption for COLMAP to work on these instances. It can easily fail even if there is a slight variation in geometry and appearance, which raises significant concern on the method's robustness to instance variations. - Once the multiple instances are registered, the rest of the pipeline seems to be a standard inverse rendering task, and is highly inspired from existing work, which I do not find sufficiently interesting. ### W2 - A special case of the rose paper [66] - Line 111: "independently and concurrently, Zhang et al. [66]" - NeurIPS has a two-month rule for claiming concurrent work, including arXiv papers. [66] appeared on arXiv in Dec 2022, which is almost half a year from the NeurIPS submission deadline. It might be difficult to claim concurrent work. - Essential, duplicates are an extreme form of the definition of similarity in [66]. The instance variation is ultimately what makes [66] interesting, as it breaks the assumption for a standard multi-view stereo approach, hence the generative modeling. - If the proposed method can handle instance variations well, such as different rose instances, without assuming a prior pose distribution, it will make a much more compelling argument. ### Other comments - I suppose the material network is inherited from [65], as the "latent code" $\mathbf{\rho}$ and its regularizer are the same as [65]. But this is not mentioned in the paper.
Questions
It might be hard to change my opinion on this paper, as most of the technical aspects are clear to me. Rather, this is a slight trivial task, especially given [66] has demonstrated results on a more generic case. Yet, I'm still open to objection.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
1 poor
Limitations
The paper attributes the challenge of the modeling instance variation to the shape/material model, but the fundamental challenge is rather the pose estimation with COLMAP to begin with. COLMAP generally fails terribly with deformable objects - also the reason that recent deformable NeRFs typically mask out deformable objects running COLMAP to register cameras.
Summary
This paper tackles the inverse graphics task of predicting the geometry, material and illumination from a single image containing multiple identical objects. The key insight is to leverage the multiple instances depicted in the image to frame this single-view multi-object reconstruction problem into a better constrained multi-view single-object reconstruction problem. Specifically, the proposed approach (dubbed SfD) first identify and crop the instances in the image, then estimate their 6DoF poses using SfM to create a set of calibrated multi-view images. Finally, a geometric reconstruction module based on prior MVS works is optimized using the resulting multi-views. The framework is validated through experiments conducted on a custom dataset dubbed Dup and introduced in this work.
Strengths
S1. Leveraging duplicated instances to frame a single-view multi-object reconstruction problem into a multi-view single-object one is an interesting idea. Besides, this idea is rather novel; it has only been studied by the concurrent CVPR'23 work of [66] S2. Experiment section looks strong; the compared methods are relevant and the reported performances are better than prior works S3. The presentation is very clear and well written. It is easy to walk through the method formulation, which does not contain any major technical flaw. Figures are neat, well designed, and help the understanding.
Weaknesses
**W1. New problem with low applicability** Although conceptually interesting, this work is tackling a new problem which by essence is very limited in terms of applicability to real-case scenarios, thus questioning its usefulness. Indeed, I see very few cases where the approach developed here could be successful, and quite a lot of failure cases: for example, I think the proposed approach would fail on most of the examples advertised in Figure 2 (see W2 for details). **W2. Lack of analysis** The current presentation lacks some crucial analyses to better understand the proposed approach. In particular, I would expect experiments outlining: - the contribution of each key component (quantitative ablation study): each of the 6 loss terms, rotation aware data augmentation, etc. - its robustness to real data: in particular not exact duplicates like leaves of a tree - its robustness to noisy input: an image with duplicates + some other objects (e.g. chairs and tables in Fig. 2) or an image with only one-side instance views (e.g. cars in Fig. 2) - its robustness to high number of duplicates (e.g. 100/1000 objects like the pile of screws in Fig. 2) - the failure cases in the general setting (e.g., from the video in the supmat, it seems the geometry of the coke can is actually not so great, why?) **W3. Highly engineered method** In addition to the low applicability of the formulated problem, the proposed approach is heavily engineered (6 loss terms, two-level approach including an optimization level with multiple stages, etc). Such amount of engineering typically makes it hard to apply the method to other cases or even to modify and build on top of it to improve the method.
Questions
- Table 1 is overflowing and appears way too early in the presentation (harms the overall readability) - L142: for completeness, I would suggest formulating the changes in the intrinsic matrix when cropping is performed (in the supplementary material) - Eq (4) and (5): there is no scaling factors to weight the different loss terms? - in general, the paper looks crowed because spaces have been modified; I think the presentation would gain in readability by leveraging the extra page to make it breath a bit more (especially true for the tables)
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
There is a small limitation section, which I think should be more detailed and illustrated to help the reader better understand the proposed method (see W2)
Summary
The paper presents a method for reconstructing the geometry, material, and illumination of an object using as input an image containing multiple copies of the object. The method leverages the appearance of multiple instances of an object in a single image to essentially create a multi-view supervision signal. The method first segments the different instances of the object in the image, then performs 6DoF pose estimation followed by a structure from motion pipeline to create an "artificial" multi-view setup. The object geometry and appearance is modeled with Neural Fields following NeuS. The whole pipeline is supervised using a photometric loss between the renderings and the original image.
Strengths
1. I find this paper particularly interesting. I really like the idea of formulating a duality between multiple copies of an object in a single image and multiple views of a single object. It is a very nice example of thinking outside the box. 2. I enjoyed reading the paper. It is well-written and easy to follow. 3. The individual components used are properly justified and the engineering behind the method is also really good. It's a lot of individual components (segmentation, pose estimation, SfM, geometry and light representation) stitched together and it requires a lot of effort to make them work. 4. The qualitative and quantitative results (Tables 2 & 3) are very good. I would like to see this paper get accepted.
Weaknesses
1. I can't find any particularly important weakness. One issue is that the problem the paper attempts to solve is not something someone will encounter very often in real-world scenarios, but it's definitely very interesting from a scientific perspective.
Questions
I would like to see what are some failure cases of the method. I couldn't find any in the main paper or supplementary material.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
The paper has a decent discussion on the limitations of the proposed method. I could not identify any immediate potential societal impact of this work.
I read the other reviews and the rebuttal. It seems that the reviewers are split, with the paper getting 5 different ratings (SA, A, WA, BR, R). I understand the objections that the reviewers raised about the potential usefulness of the method; it's hard to find many real-world examples where one would encounter this scenario. Also the reviewers highlighted (including me) that the method is mainly about engineering and putting together a lot of already well-known components. I also understand the limitations that other reviewers highlighted, such as that COLMAP fails in the presence of non-rigid deformations, but this is not immediately applicable in the use cases presented in this paper, because most objects as Reviewer 4hmW noted are industrially manufactured, so there is little variation between instances. There are other papers that have explored somewhat constrained or "artificial" setups such as: - Learning the Depths of Moving People by Watching Frozen People (CVPR 2019) - Reconstructing 3D Human Pose by Watching Humans in the Mirror (CVPR 2021) - Through-Wall Human Pose Estimation Using Radio Signals (CVPR 2018) Overall still believe this is an interesting problem from a scientific perspective and I would like to see the paper get accepted.
We appreciate the reviewer's thorough review of our paper and the subsequent responses to our rebuttals. We would like to thank the reviewer for recognizing the positive contributions of our work from a scientific perspective. We will also cite the three papers the reviewer mentioned and discuss them in the final version.
Summary
This paper presents "Structure from Duplicates" (SfD), a novel inverse graphics framework introduced to reconstruct the 3D structure, material, and illumination of multiple identical objects from a single image. The key steps include identifying these duplicate objects in an image and estimating their 6DoF pose. Then, the model applies an inverse graphics pipeline to deduce information about the objects' shape, material, and the lighting of the scene, while taking into account that these objects share the same geometry and material properties.
Strengths
In my opinion, the key strengths of the proposed inverse rendering method are: - The method outperforms baseline techniques in single image multi-object setup and multi-view inverse rendering scenarios, as shown in the experiments. - The approach can leverage duplicate objects to constrain underlying geometry and better disentangle the effects of lighting from materials. - It can be extended to multi-view setups. - The model makes effective use of duplicate objects, with accuracy improving as the number of duplicates increases. - It performs well with real-world data, achieving comparable performance to multi-view baselines when trained using only a single view. - The model supports various scene edits - once the material and geometry of the objects are recovered, along with the scene's illumination, it can faithfully relight existing objects, edit their materials, and insert new objects into the environment.
Weaknesses
The potential weaknesses seem to be: - Dependence on Similarity. A significant limitation is that the instances in each image need to be nearly identical; the method struggles when there are substantial deformations between different objects. - Pose Estimation Errors. The model currently requires accurate 6 DoF poses as input and keeps the poses fixed, which may limit its applicability in certain scenarios. - Potential Segmentation Errors. SfD begins by identifying multiple instances of an object within an image. This is prone to errors when the segmentation is inaccurate which is more of a problem on glossy and specular objects.
Questions
- How would the model perform when applied to real-world scenarios with more noise, variability, and less regularity in object structures? - Could you discuss any strategies for mitigating the potential overfitting that might occur if you increase the capacity of the model to allow for instance-wise variations? - Could you elaborate on the potential of integrating your approach with BARF. What kind of improvements or challenges do you expect?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Limitations are discussed above in the weaknesses section. Overall I am on the fence due to the limitation of the proposed approach, which is highly dependent on the similarity of the multi-objects in a single image.
**Additional results on ablation studies** I still wonder if more than 10 objects (results in the main paper) would improve the reconstruction results significantly and if there is a number of objects that can count as an upper bound for reconstruction improvements. Do you have any insights on that? **Representation agnostic method** Clarifying, that the presented approach is agnostic of the 3D representation and can be improved with more advanced models makes sense. Looking forward to the Neuralangelo [2] results!
To understand how the performance of our model scales with respect to the number of instances, we increase the number of objects to 15 and 20. Due to time constraint, we only test on the new scene of boxes. The preliminary results are shown below. We will include more comprehensive investigation (e.g., more than 20+ objects, more categories) in the final version. Our current hypothesis is that there do exist an "sweet spot" number due to the accumulated 6 DoF pose error brought by heavier occlusion, the limited capacity of the visibility field, and limited image resolution. | | Albedo | Roughness | Env Light | Relight | Env Light | Normal error | |-----|---------|-----------|-----------|--------------|-----------|---------------| | Num | PSNR ↑ | MSE ↓ | MSE ↓ | PSNR ↑ | MSE ↓ | °↓ | | 5 | 20.053 | 0.086 | 0.086 | 19.229 | 0.086 | 3.329 | | 10 | 19.517 | 0.095 | 0.095 | 19.021 | 0.095 | 2.578 | | 15 | 20.522 | 0.093 | 0.093 | 18.769 | 0.093 | 2.433 | | 20 | 22.530 | 0.077 | 0.077 | 19.560 | 0.077 | 2.279 |
Thank you for your time investigating my questions and the provided results, those are really insightful. I encourage the authors to add all additional results and discussions to the final manuscript. And after reading all detailed discussions I will keep my rating and recommend to accept this paper.
Response to the rebuttal
I appreciate the immense effort the authors have put in the rebuttal. It certainly sheds more light in the motivation of the work and the limits of the model. However, I still disagree with the major claims in the paper and in the rebuttal. --- **Regarding the technical contributions, I respectfully disagree that this is "an interesting _scientific_ attempt" and that this will "make inverse graphics/3D reconstruction more robust and work under more extreme scenarios".** I certainly acknowledge that from an engineering perspective, the pipeline is not trivial. Or in fact, nothing is trivial when it comes engineering. But from a scientific perspective, what a "trivial" task means to me is: when one describes the problem, we already know in principle it's going to work. This is an example of such a problem: *given multiple duplicate copies seen from various viewpoints*, we know COLMAP works and the rest of the inverse-rendering pipeline is [65]. Therefore, the results do not appear surprising to me. When current structure from motion methods / COLMAP fail, the proposed pipeline also fails the same way. I do not see a clear and compelling argument on how this will "make inverse graphics/3D reconstruction more robust". The only new scientific insight I gained from this paper is that in some cases when COLMAP fails due to the number of views being too small, in-plane rotation augmentations may help. But this is less of a significant contribution and is not the primary claim of the paper. --- **Regarding the novelty of the problem, I also disagree with the claim that "this is the first effort to conduct structure from motion from a single image" and the general notion that this brings a spanking new perspective to the community.** The rebuttal puts a lot of emphasis on the novelty of this task and "formulating a duality between multiple copies of an object in a single image and multiple views of a single object". The statement of the problem of "recovering object intrinsics and environmental extrinsics from a single image without pre-trained priors" is exactly the message of [66]. But nor is [66] the first one to look at this (and it thus focuses on recovering a distribution over the instance variations). One common venue where this task of structure from motion on multiple instances in a single image has been explored is robotics [A, B]. I would agree with the statement with additional conditions: "this is the first effort to conduct structure from motion from a single image of (near)-identical objects without any assumptions on the pose distribution or depth", and the novelty of the proposed problem should be assessed in light of these conditions. --- **References**: [A] Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd. CVPR 2016. [B] Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects. CoRL 2019.
Further comments on the new "crane" and "paint bottles" results
**The new results on the "cranes" and "paint bottles" reveals issues** When the authors ascertain that the method "produces satisfactory reconstruction", they only show **masked** images rendered from the **input viewpoint**. As all instances have been registered onto the same 3D model, why not visualize the full 3D models rendered: (1) onto the input instances with the estimated poses **without masking**; and (2) from **various viewpoints**? This is an issue with all visualizations in the main paper as well, although in the paper examples with perfect duplicates, it is more obvious the results are reasonable. However, in these new examples in the rebuttal where the inputs are corrupted by instance variations and erroneous poses, the reconstruction significantly deteriorates and it is much less obvious how well they actually are in 3D. Most critically, the `crane` results seem to reveal poor correspondences due to inaccurate poses and shapes, and the `paint bottles` results also show that the model suffers from and fails to improve the inaccurate poses. Masking out the renderings (not rendering the full object) and rendering only from the input views (not novel views) conceals these failures. Further questions about the `crane` results: - First of all, did COLMAP succeed? It would be quite surprising if it did. And if not, how are the poses initialized? - How exactly is the instance variation implemented? Is it only allowed in the geometry, not in the appearance? If so, why is the `diffuse` visualizations look so different for each instance? - The bounding boxes are confusing; it is hard to tell the actual orientations of each instance. It would be much more insightful if the shape, deformation and poses are better visualized so that one can tell the whether the actual canonical reconstruction and the correspondences are reasonable. For instance, rendering reconstruction of one instance onto another, and from other views. Further questions about the `paint bottles` results: - Why not reporting numbers on the recovered pose, shape and material, instead of visual results, since the perfect GT is available from the synthetic data? - But more critically, where do the initial noisy poses come from? By perturbing the GT poses or from COLMAP? Does COLMAP work on these cluttered bottles?
regarding novelty (in response to Reviewer 4hmW)
We kindly disagree with the reviewer's argument on contributions. - *“We already know in principle it's going to work”*: Contrary to the reviewer's perception, the problem is, in fact, far from trivial. We were quite surprised when COLMAP, even with the incorporation of state-of-the-art feature matching techniques like SuperGlue and LoFTR, fails to estimate the 6 DoF poses. Additionally, since the relationship between illumination and the object differs in our new setting, we need a careful reformulation and reimplementation of the inverse graphics problem. - *“The results do not appear surprising to me”*: We would like to remind the reviewer that novelty is not equivalent to surprise [A]. As pointed out by other reviewers (Reviewer sRgJ, qVSc, vbZQ), we firmly believe that our work introduces an intriguing new setting and a principled way to leverage rich multi-view cues from a single image. [A] Quote from “Novelty in science”: https://perceiving-systems.blog/en/news/novelty-in-science
Improvement over COLMAP poses
(I apologize there is no threading on OpenReview. It might be difficult to keep track of conversations. This is the response to the "regarding novelty" comment.) I completely agree with the authors that "We were quite surprised when COLMAP, even with the incorporation of state-of-the-art feature matching techniques like SuperGlue and LoFTR, fails to estimate the 6 DoF poses." And in the cases when COLMAP fails, simply adding in-plane rotation augmentation resolves the issue? How robust is this observation? Does it succeed most of the time? Except in the rebuttal, there is no further pose refinement in the original paper. What else in the original pipeline (except the in-plane rotation augmentation) that further improve the robustness of the pose estimation? I think this is exactly my point in the my first comment above: "given multiple duplicate copies seen from various viewpoints, we know COLMAP works... When current structure from motion methods / COLMAP fail, the proposed pipeline also fails the same way." Except the only insight I gained: "in some cases when COLMAP fails due to the number of views being too small, in-plane rotation augmentations may help." In terms of pose refinement with the full inverse rendering pipeline in the `paint bottles` example in the rebuttal, given only the masked normal renderings from the input viewpoints in Fig. 3, it does not look very conclusive to me whether the poses were improved. In particular, the normals `with BARF` pose optimization look worse/flatter than the normals `with c2f` without further pose optimization. It would be interesting to see some numerical evaluation on both pose and geometry (eg, angle deviations of the normals), given the synthetic GT (as the authors are already working on this).
regarding results on "crane" and "paint bottles" (in response to Reviewer 4hmW)
The reviewer misunderstands our results. **"Various viewpoints"**: Given the variations of object poses (wrt the camera) in `crane` and in `paint bottles`, we have already showcased various viewpoints of the objects within a single image. This is a distinct feature of our setup where we have multiple instances, each with different poses. If only a portion of our object is rendered precisely, certain instances will present rendering artifacts. That being said, we are more than happy to provide a video that displays the recovered object rendered from continuously varying poses and lighting conditions in the final version. **"Masking"**: Since we adopt masking loss during training, the difference between employing masking or not during visualization is negligible. We are happy to include the visualization without masking in the final version. **"Novel views"**: Following common practices in the inverse graphics community, we showed the inferred shape, normals, albedo, and renderings aligned with the input observations. Additionally, we supplied videos that are well-suited for observing novel views and relighting effects (please see our supplementary video for reference). We will more than happy to include novel view images in the final version.
on the "crane" and "paint bottles" results
I do not understand how I misunderstand the results. I'll rephrase my comments: 1. **The visualizations are rendered from only the input viewpoints.** Of course, I can see these input viewpoints are different. What we need to see, for any 3D reconstruction / inverse rendering methods, is the reconstructed results from novel viewpoints different from the observed input viewpoints. Or if the authors do not care about the novel viewpoints at all (by the argument of "following common practices in the inverse graphics community"), rendering with novel lighting conditions can be useful too. The normals and albedo from the input viewpoints in Fig. 1 and Fig. 3 don't look reasonable to me either. 2. **The renderings are masked**. This looks also very confusing to me. The authors claim "Since we adopt masking loss during training, the difference between employing masking or not during visualization is negligible". - I can imagine this being the case in the `crane` example, as there are instance geometry and appearance variations being optimized. But what becomes more critical here is whether the correspondences are preserved, ie, left wings match left wings, head matches head. The bounding boxes did not help discern this. - In the `paint bottles` example, it is hard for me to imagine this is the case. The texture and normals clearly show incorrect correspondences. If the same canonical model is able to fit all masks perfectly, the only explanation I can think of is the volume rendering in NeuS overfitting each instance view using translucent volumes / floaters, but this will be a failure case that we should avoid for inverse rendering. This can be seen clearly from novel views, rotation animations or extracted meshes. Unfortunately no more visualizations can be shared at this point. **Correspondences on the `cranes`** It is very helpful to clarify "both color and geometry fields can be adjusted". So essentially, the model has sufficient capacity to overfit each instance. This makes the question on the correspondences even more critical. First, to verify the relative poses are reasonable, may I suggest one way of reporting them here? I am not sure this is within the limit of this discussion cycle. The authors can totally ignore this if it is a nonsense request. Is it possible to give a list of rotation angles for each instance, in the format of `(u, v): (rx, ry, rz)`, where `(u, v)` indicates the center of the instance mask in the image normalized to `(0, 1)`, and `(rx, ry, rz)` indicates Eular angles (in any rotation order, say `ZYX`, as z is the dominant axis of rotation here), assuming right-hand coordinate system with camera pointing towards `-z` and `+y` being upright? - To make things easier to interpret, one could fixate the pose of one reference instance and only report the relative poses with respect to that reference instance. - It would also be very helpful to report the poses estimated using different procedures, if any: - (a) using COLMAP on the original masked patches - (b) using COLMAP with in-plane rotation augmentation - (c) if there is any additional customized BA that the authors further apply, report those results as well I think this will help us better understand the correspondences. As for the the current inverse rendering visualizations in Fig. 1, the normals and albedo look very coarse to me, very similar to the quality that SIRFS can produce by optimizing on individual instance patches. It is not convincing to me they would look reasonable in 3D.
**Novel views** We invite the reviewers to see the supp video for both novel poses and novel lighting for our paper’s datasets. For the new examples in rebuttal, we will add novel views in the final paper / supp video. **Additional differences in execution pose estimation using COLMAP** We use a customized procedure to transform multiple relative pairwise poses into an initialized global rotation angle for each instance. This is done while accounting for potential conflicts among each pairwise estimate. We have implemented this using Scipy’s BFGS minimizer. This added detail ensures its proper integration into the standard BA pipeline. Further specifics will be provided in the paper. To what extent does plane rotation augmentation increase the robustness of BA in colmap. Vanilla COLMAP often fails to reconstruct the instances. With the help of in-plane rotation augmentation, we can greatly improve the performance (e.g., from failure to success). That being said, for low-texture, low-resolution scenes where pixels are not distinct, our method may still suffer. **Poses estimated result using different procedures** We will report this in the final version. **Performance on 70 paint bottles** We evaluated the 6DoF pose errors for the 70-paint bottles dataset. The results indicate that in-plane rotation augmentation significantly improves results (reducing the median error from 77 deg to 11 deg). It’s also important to mention that due to extensive occlusions, the 70 instances present greater challenges than cases with fewer instances, leading to an increased absolute error and outlier poses. We will elaborate on these findings in the paper. That said, it’s noteworthy that in inverse graphics, similar to standard MVS 3D reconstruction pipelines, one can choose instances with a higher confident 6-DoF pose using # of inliers wrt inferred poses. | | median(mean) dr | median(mean) dt | |----------------------------|------------|------------| | w/o augmentation | 77.1° (65.62°) | 63.98° (52.01°) | | with augmentation | 11.7° (38.35°) | 10.53° (25.75°) | **Correspondences on the cranes** Due to time constraints, we may not complete the Euler angles analysis as recommended by the reviewer. However, we did examine the resulting SFM point cloud, which captures the coarse geometry of the Crane, indicating the viability of the 6DoF poses. In this scenario, reliable correspondences mainly occur at corner and edge points, such as the crane’s beak, tail, etc. Additionally, we will visualize both GT and inferred poses for synthetic cases. Please note that we intend to release all the code and results, providing a more clear understanding of the quality of correspondences and poses. **Visualization of the mask** We will visualize the object without the mask in the final version.
further questions on "crane" and "paint bottles" (in response to Reviewer 4hmW)
**"Did COLMAP succeed?"**: We recognize that even after implementing rotation augmentation, incorrect correspondences still exists in the crane image (errors in feature correspondences are simply unavoidable). As with standard SFM, bundle adjustment can compensate for these correspondence errors, yielding a reliable initial 6-DOF pose estimate and a sparse point cloud. We will present the BA results in the supplementary material. Bundle adjustment completes in 10.56 seconds, with an average of 47.3 inliers, 191 triangulated points, and the geometric energy values converge to 0.995363 pixels after ~50 iterations for the Crane. **"How exactly is the instance variation implemented?"**: We remove the strictly shared 3D representation constraints, allowing per-instance optimization of the last 4 layers of the NeuS model parameters. Both color and geometry fields can be adjusted. We employ regularization to ensure that the parameters don't deviate excessively. **"The bounding boxes are confusing"**: We agree with the reviewer that the pose visualization can be improved. We will follow the reviewer's suggestion and incorporate more visual content in the final version. **"Where do the initial noisy poses come from?"**: They are from bundle adjustment. In highly cluttered scenes with significant occlusions, BA may not infer the high-confident 6-DoF pose for certain instances, as we have highlighted. However, achieving a perfect initial pose isn't necessary, thanks to the later refinement, as shown in the rebuttal. **Numbers on paint bottles**: We are working on them. We will update this thread as soon as we finish all the evaluations.
Difference to [65] (in response to Reviewer 4hmW)
**Contributions from the inverse graphics perspective:** We have expanded the scope of the inverse graphics family by introducing a novel "single-view duplicate objects" setting. While [65] follows the traditional multi-view single-instance (M-S) framework, our focus lies in a single-view multi-instance (S-M) scenario. It's important to note that these two problem formulations are very distinct. Specifically, our proposed setting brings forth several advantages compared to the existing setup. In the conventional configuration, the relative poses between the objects and the lighting sources remain fixed. This poses a significant challenge when attempting to distinguish an object's albedo from its illumination. For instance, it is difficult to tell whether it is a yellow directional light on the left illuminating a white ball or it is a ball that is half-yellow and half-white. In contrast, in our setup, the relative poses between the object and the light sources differ for each "virtual" view (*e.g.,* the env light is rotating in SO(3) space). This allows us to more effectively recover geometry, materials, shadow, and illumination while utilizing the same number of instance observations compared to [65]. Experimental results supporting this claim can be found in Table 1. **Key differences in execution:** - *Backbone*: In our approach, we utilize NeuS as our neural surface model; and for the visibility field, we opt for Siren instead of ReLU. (We will highlight the distinction in the final version.) - *Metallic*: We additionally reason for metallic materials (refer to Line 209). - *MLP distillation*: We distill the geometry MLP into a smaller one for fast classification. - *Self-occlusion and Inter-occlusion*: Since we have multiple instances in our setup, it is essential to model both inter-object self-casted shadows and inter-object occlusions. Our model goes beyond simple object-centric representation. **Remark on “Colmap”:** We stress that our pose estimation process IS NOT just a simple application of Colmap. Merely employing Colmap will not be sufficient, as we have pointed out in the paper (Line 152). Part of our contribution lies in how we jointly reason the 6 DoF poses with a carefully designed matching scheme and how to integrate it into the BA framework. We only use Colmap for its BA optimization. [65] Modeling indirect illumination for inverse rendering
Thanks for summarizing the key differences from [65]
Thank you for summarizing the **Key differences in execution** from [65]. This is very helpful! Regarding the pose estimation using COLMAP, the "carefully designed matching scheme" consists of adding in-plane rotation augmentations (which is a very useful observation) and potentially also masking out the instances with a pretrained segmentation model, correct? If there are additional key differences in execution, it would be super helpful if the authors can also summarize them. And regarding the "single-view duplicate" vs "multi-view" settings, we are on the same page. I understand perfectly the differences between the two, and I think we have sufficient discussions on this.
regarding related work [A, B] (in response to Reviewer 4hmW)
We thank the reviewer for pointing out the two works on joint 6-DOF pose estimation [A, B]. We will carefully investigate them and refine our positioning statement in the final version. We, however, note that the setups and the goals are very different (*e.g.*, RGB (ours) vs RGB-D, inverse graphics/3D reconstruction vs pose estimation). [A] Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd. CVPR 2016. [B] Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects. CoRL 2019.
Thanks for the detailed responses
I would like to thank the authors for preparing the detailed responses. I have posted my response in the general thread above as I was called out there. One additional comment: it would be very helpful if the authors could summarize, once the pose is registered using COLMAP, what are the key differences / technical contributions between the proposed inverse rendering pipeline and [65]?
Difference to [65]
**Contributions from the inverse graphics perspective:** We have expanded the scope of the inverse graphics family by introducing a novel "single-view duplicate objects" setting. While [65] follows the traditional multi-view single-instance (M-S) framework, our focus lies in a single-view multi-instance (S-M) scenario. It's important to note that these two problem formulations are very distinct. Specifically, our proposed setting brings forth several advantages compared to the existing setup. In the conventional configuration, the relative poses between the objects and the lighting sources remain fixed. This poses a significant challenge when attempting to distinguish an object's albedo from its illumination. For instance, it is difficult to tell whether it is a yellow directional light on the left illuminating a white ball or it is a ball that is half-yellow and half-white. In contrast, in our setup, the relative poses between the object and the light sources differ for each "virtual" view (*e.g.,* the env light is rotating in SO(3) space). This allows us to more effectively recover geometry, materials, shadow, and illumination while utilizing the same number of instance observations compared to [65]. Experimental results supporting this claim can be found in Table 1. [65] Modeling indirect illumination for inverse rendering **Key differences in execution:** - *Backbone*: In our approach, we utilize NeuS as our neural surface model; and for the visibility field, we opt for Siren instead of ReLU. (We will highlight the distinction in the final version.) - *Metallic*: We additionally reason for metallic materials (refer to Line 209). - *MLP distillation*: We distill the geometry MLP into a smaller one for fast classification. - *Self-occlusion and Inter-occlusion*: Since we have multiple instances in our setup, it is essential to model both inter-object self-casted shadows and inter-object occlusions. Our model goes beyond simple object-centric representation. **Remark on “Colmap”:** We stress that our pose estimation process IS NOT just a simple application of Colmap. Merely employing Colmap will not be sufficient, as we have pointed out in the paper (Line 152). Part of our contribution lies in how we jointly reason the 6 DoF poses with a carefully designed matching scheme and how to integrate it into the BA framework. We only use Colmap for its BA optimization.
Thanks for the rebuttal
I thank the authors for the detailed rebuttal, which I have read along with the other reviews. In general, it addresses most of my concerns and I still think this paper presents valuable insights for our community. I strongly encourage the authors to include these discussions in the final version to further increase its quality. I will keep my rating and recommend to accept this paper.
Thanks for your comments; we will discuss all the results in detail in the final version.
Decision
Accept (poster)