NU-MCC: Multiview Compressive Coding with Neighborhood Decoder and Repulsive UDF

Summary

The paper studies the problem of shape reconstruction from single-view RGBD images and addresses two key limitations in a major prior work, MCC: 1) inefficient decoder of quadratic time complexities, and 2) suboptimal implicit representation for pointclouds. To improve computational efficiency, the authors propose an anchor-based neighborhood aggregation scheme to downsample the seen points. They also propose a way to sample features from the high-resolution seen points directly that improves the modeling of the fine surface details. On the other hand, instead of using occupancy representation, UDF is adapted in this paper for higher sample efficiency and more accurate surface modeling. To further resolve the problem of point concentration, the authors add a concentration penalty in the energy function. The authors show evaluation of their model on CO3D and HyperSim (supplement) and demonstrate state-of-the-art performance over MCC. An ablation of different designs is also included.

Strengths

1. The limitation of the prior work (MCC) is well analyzed in this paper. 2. The design of the repulsive UDF makes sense and looks novel. 3. The paper is relatively easy to follow.

Weaknesses

1. Missing quantitative comparison to PointTr. Based on my understanding, PointTr leverages a similar approach of anchor+neighbor aggregation. Given the results in MCC (Table 1(f)), PointTr performs significantly worse than MCC's decoder, while the finding in this paper shows similar idea to PointTr actually leads to better performance. A detailed discussion on this seemingly contradictory finding, as well as an apple-to-apple comparison to a baseline with PointTr's feature aggregation method would be very helpful. 2. It is not clear where the major speedup of the inference actually comes from. In L116, it is stated that MCC's decoder is quadratic in the number of query points. However, if my understanding is correct, this is more of an implementation/engineering issue. In fact, query points do not communicate with each other in both works, so the effective time complexity should be linear on the number of query points if implemented well. My worry here is that if the authors implement MCC in some naive quadratic way while implement their method in the effective linear way, the main speedup can actually come from this implementation difference rather than the claimed novelty. It is very important to clarify on this. 3. It would have been very helpful to break down CD into accuracy and completeness. The RepUDF representation makes sense in terms of completeness improvement. But I think it is very important to see whether it helps accuracy as well. 4. The usage of nearest points to improve fine details makes sense for seen surface, but does not make much sense for unseen surface. Is it possible to evaluate the model's performance on seen and unseen surface separately? If it only improves RGB/surface reconstruction quality on seen surface, I wonder whether a simple baseline that directly overwrite the seen part of the reconstruction with seen points would actually perform similarly to the proposed method. For example, for every queried point in the reconstruction, if there are any seen points that are close enough to this queried point, we can replace the queried point with the seen point.

Questions

Please see the weakness for my concerns and questions. My final rating will be highly dependent on authors' response to these questions.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The limitations of the work have been adequately discussed.

Summary

This paper approaches the task of 3D reconstruction given a single RGBD image. It builds off of recent SOTA approach MCC, adding four main contributions: (1) applying attention on only anchor points instead of each query point and using weighted interpolation, (2) using linear projection for “fine” features at each input point (3) replacing occupancy representation with UDF, (4) adding a “repulsive” component to UDF. Experiments on CO3D-v2 validation set show quantitatively (1) provides significant speedup, (2) improves fidelity, (3+4) improve fidelity further. Qualitative and quantitative results show favorable performance vs. MCC.

Strengths

Method’s contributions are strong: - The quadratic nature of attention means carefully constructing queries can yield efficiency. This combined with the insight of extracting fine detail through linear projection is a nice combination. - Repulsive UDF is intuitive and positive contribution that is clearly better than occupancy. In the well-studied field of reconstruction, this is a helpful insight. - In the quantitative result that is reported, results are compelling. Ablations show (1) anchors provide significant speedup, (2) fine features improve fidelity, (3+4) Rep. UDF improves fidelity further. A qualitative example is convincing the repulsive UDF is meaningfully better than UDF. Writing is generally excellent. - For the most part, one could reproduce most of the method and experiments from reading the paper. - Technical details are well-motivated through a thorough comparison with MCC. The reader feels like an expert in these methods after reading the paper.

Weaknesses

Edit: after reading the rebuttal and other reviewer comments, the lack of comparisons weakness is remedied. Lack of quantitative comparisons and metrics; existing evaluation is inconsistent with prior work. - The key contributions (1-4 from summary) are validated in only one quantitative result (and one quantitative result in Supplemental). Results on CO3D are not broken down into seen and unseen categories, which would be helpful. - Why not include more qualitative results in Supplemental? Two examples on e.g. ImageNet are not convincing, especially when no quantitative comparison is reported. - The existing quantitative results do not follow the evaluation protocol of prior work (MCC) -- the paper appears not to report the accuracy, completeness, absolute error or MSE metrics of MCC. The numbers reported for MCC following F1-score metric do not match the prior work – presumably this is because numbers are reported on the validation split, while MCC uses the test split? This is a strange choice. Why are F1 metrics on Hypersim are significantly different from MCC? - No comparison or at the very least reference to PointE is surprising. This relates to why I gave a 3 instead of 4 for presentation: it is strange the paper builds off MCC (unpublished, arXiv Jan 23), while ignoring entirely Point-E (unpublished, arXiv Dec 22). While not comparing to unpublished work is acceptable, it is strange to not use this work which other works (e.g. 3DFuse, Zero 123) have found outperforms MCC.

Questions

To further strengthen the contributions of this paper, a more thorough ablation study upon contributions could really help. For instance: - Does increasing the resolution of fine features help, as claimed in L187? - How sensitive are results to the number of anchors or anchor radius? - Do details of repulsive UDF sampling matter? Other questions: - I don’t follow the section on bootstrapping the anchor predictor (L131). How is global token z0 used exactly? Where does z0 come from? - Are the same weights used for both coarse and fine, given they are concatenated and in practice the same number of coarse and fine weights are used at inference? - Figure 3 could use a key, as currently one has to look at the text to understand int - Figures 2-3 have text that is very small, making it a bit hard to read - L147: “depends” should be “depend”

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Yes

Summary

The paper introduces a novel method, NU-MCC, for reconstructing 3D objects from single-view RGB-D images. The proposed approach builds upon the existing MCC method by incorporating two important advancements. Firstly, it introduces a neighborhood decoder to efficiently manage a large number of query points, which helps to reduce computational complexity and improve processing speed. Secondly, it adopts a Repulsive unsigned distance function (Repulsive UDF) to enhance the quality of surface reconstruction, resulting in more accurate and detailed reconstructions. The authors have conducted extensive experiments on multiple datasets to evaluate the effectiveness of their proposed approach. The results demonstrate that NU-MCC outperforms the existing state-of-the-art methods, achieving higher accuracy and better quality reconstructions.

Strengths

This paper identifies and addresses two major limitations of the state-of-the-art MCC method. The first limitation is related to the Transformer decoder, which struggles to handle a large number of query points efficiently. The second limitation is associated with the inability of the 3D representation to recover high-fidelity details accurately. To address these limitations, the authors propose a new method called NU-MCC that significantly improves the quality of 3D object reconstruction. The proposed method employs a set of anchor points to provide global coarse structure prediction and utilizes the Neighborhood decoder to accelerate inference speed while enabling the exploitation of finer-scale visual features for improved recovery of 3D textures. Additionally, the Repulsive Unsigned Distance Function (UDF) is utilized to replace the occupancy field for more accurate surface representation, outperforming the traditional UDF in this problem. The effectiveness of the proposed NU-MCC method is demonstrated through experimentation on multiple datasets. Specifically, the method quantitatively outperforms MCC by 9.7% in terms of F1-score on the CO3D-v2 dataset. Moreover, it showcases its superior performance qualitatively in zero-shot settings such as iPhone RGB-D capture, AI-generated images, and ImageNet. Overall, this paper provides valuable discussion into the challenges associated with 3D object reconstruction and demonstrates the efficacy of the proposed NU-MCC method in overcoming these challenges.

Weaknesses

Overall, I find the method to be promising, but there are several concerns that need to be addressed regarding the evaluation section. Firstly, I believe the analysis of the effects of different anchor point numbers is missing. Given that the use of anchor points is one of the key designs of the method, it is important to understand their effect on the performance of the model. Therefore, it would be beneficial to conduct experiments analyzing the effects of different anchor point numbers to improve the evaluation section. Secondly, while the repulsive UDF is another important design of the method, there are no quantitative results provided to ablate its influence. Although a single visual comparison is presented in Fig. 5, more experimental results are needed to justify this design and its impact on the overall performance of the model. Thirdly, there seems to be some inconsistency with regards to the resolution used for training and testing. While the supplementary material states that the model trained with 128128 image resolution can be directly applied to 256256 image inputs, the main paper suggests that the model is trained with a 256256 resolution. It would be better to clarify these discrepancies and consider using a higher resolution such as 512512 for testing. Furthermore, no video results are shown to check the 3D reconstruction results, which could provide a more comprehensive evaluation of the proposed method. Lastly, there is a need to explain the design philosophy behind the aggregation weight estimation in Equation 2. It is unclear why the sum operation is used when the input to the network is the sum of three features. Therefore, providing more insights into the selection of weights and the reasoning behind it would enhance the clarity of the method. In conclusion, while the method shows promise, the issues outlined above need to be addressed to improve the evaluation section and to strengthen the overall contribution of the paper.

Questions

- Analysis of using different numbers of anchor points. - More results (both quantitative and qualitative) to justify the repulsive UDF representation. - Higher image resolution for testing. - Video results for the reconstruction. - Design of aggregation weight estimation. More details can be found in the Weaknesses part.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper embarks on the task of 3D reconstruction using single-view RGB-D inputs. Building on Multiview compressive coding (MCC), it proposes two enhancements: 1) incorporating nearest neighbors in the decoder to minimize query time, and 2) the utilization of a repulsive unsigned distance function (UDF) as the output representation. The former also introduces an intermediate representation of anchor points for efficient NN query, while the latter replaces occupancy, delivering a higher quality of 3D reconstruction.

Strengths

This paper effectively builds upon the latest research in Multiview Compressive Coding (MCC), addressing two main limitations encountered in the original study: 1) a relatively slow query speed and 2) the lack of high-fidelity details in the 3D reconstruction. In my opinion, this research is original, and I particularly like the idea of the repulsive unsigned distance function (UDF), and I believe it provides contributions to scholarly discourse in this field.

Weaknesses

The manuscript could benefit from a more in-depth explanation of the anchor predictor's impact when it is used as an intermediate representation, particularly when neighborhood points are drawn from both the anchors and the initial point cloud. Augmenting the work with an additional ablation study could aid in comprehending its effectiveness and provide more depth to the reader's understanding. In the supplementary, the quality of the scene-level reconstruction is acceptable, but it does not seem to be remarkable. It would be beneficial if the authors could identify and discuss the major barriers preventing high-fidelity scene reconstruction in their study.

Questions

1. Line. 175, the dimension of F_f is (HW)x(dx3), are all points unprojected to the 3D space? 2. Line. 254, does the number of GT points affect the model performance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The paper proposes a novel framework for generating 3D object from single view RGB-D input. Based on the previous work MCC, this work implements a neighborhood decoder that is more efficient and a repulsive udf representation that has higher quality than the previous occupancy field.

Strengths

The major improvements are sound. I appreciate the neighborhood decoder that efficiently combines coarse and fine-scale features for 3D generation. The repulsive UDF is also convincing.

Weaknesses

- What limits the generation resolution? Is it because of the point cloud quality in Co3D? Does the framework support rendering arbitrary resolution result images at inference time? - When inferencing on in-the-wild images, does the model require segmenting out the foreground mask? Fig.2 seems a bit confusing because the image involves a background. - How to determine the focal length and depth scale for in-the-wild images? - How are the camera poses set for the input at training and inference time? Do you use the ground truth camera poses provided in CO3Dv2 dataset? Are the generated 3d objects placed in their canonical pose, or are they aligned with the input camera (meaning that there's one-to-one mapping between input image and output 3d object? - Does the pipeline generate plausible 3d objects when the input depth is problematic (e.g. distorted / noisy / scaled)? For example, when working with monocular depth. - How many neighbors are used for the feature aggregation module? What's the value of $m$ in Eq. (1)?

Questions

Please refer to weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Replying to Rebuttal by Authors

The authors' response resolve most of my concerns. I have upgraded my rating from weak accept to accept. Thanks.

Reviewer Response to Author Rebuttal

After reading the rebuttal and other reviews I raise my rating to 6 - weak accept. I feel this paper should be accepted because the contributions of the method are strong, and comparison to prior work is thorough. My main concerns were surrounding evaluation details. These, e.g. F1 score or additional comparisons, were clarified or addressed in the rebuttal.

Thanks for the rebuttal!

Dear authors, A quick note to express our gratitude for your timely rebuttal. Your clarifications, responses, and the video link greatly aid the review process. Thank you, Area Chairs