Robustifying Generalizable Implicit Shape Networks with a Tunable Non-Parametric Model

Summary

The paper presents a novel shape reconstruction method (point cloud-> reconstructed implicit shape) that combines a deep shape prior and a non-parametric interpolant. Existing deep shape reconstruction methods are typically trained on synthetic datasets (to obtain full supervision) and suffer from a domain shift when applied to real examples. In addition, they are sensitive to the number of points in the input point cloud. To address scaling, the paper proposes to use Kernel Ridge Regression (KRR), a non-linear interpolant that adds the condition that the points input at test time need to lie on the output implicit surface.

Strengths

A novel method for implicit shape reconstruction that combines deep shape priors and implicit shape outputs is proposes. The technique is innovative and generates successful reconstructions on public benchmarks (ShapeNet, FAUST, and ScanNet) and outperforms three public baselines (Poco, Conv, SAP, SAC). Detailed ablation studies on the number of input points and KRR design choices (tuning, number of samples).

Weaknesses

The key weakness of the works is that the paper could be written more clearly. The methodology section has several grammatical errors and is hard to follow. It would be helpful to have more clarity and explanations for a novice reader.

Questions

The method seems to rely on a pretrained occupancy network. Does this affect the performance or expressiveness of the technique? What is the computational time of the proposed method? From Algorithm 1, it appears to be iterative. How sensitive is the method to the percentage of points that covers in the input, i.e., what if only a very partial view is provided?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The key contribution of the paper is that improves the synthetic to real gap, however, the few results on real data (sections 4.5 and 4.6) seems to indicate that performance is still significantly limited. It would help to identify specific example cases where the proposes method offers a significant advantage.

Summary

To address the poor performance of feed-forward models on noisy real data, the authors propose using test-time Kernel Ridge Regression (KRR) to fine-tune the results for implicit shape networks, specifically in the context of recovering 3D reconstruction surfaces from noisy point clouds. The method involves incorporating a feedforward network called Poco and utilizing the input point cloud and the network's approximated logits to determine the optimal KRR for representing the final implicit shape. This method improves the performance of Poco.

Strengths

1. The paper introduces an additional post-processing step, utilizing information from the input test data to enhance surface reconstruction and improve generalization ability for unseen cases. The idea is novel, as it combines a non-parametric model with a neural network to enhance the performance of the latter further. 2. The authors have evaluated their method on various datasets to assess the entire pipeline.

Weaknesses

1. The paper does not mention the overhead of the current method. Since the KRR model needs to be optimized for different point cloud inputs, it is crucial to report the optimization time in this paper. 2. Currently, the training samples for KRR consist of an equal number of noisy input and network-predicted samples. Could the authors explain the reasoning behind this decision? It would be helpful to understand why they did not include more or fewer network-predicted samples. In cases where the input data is highly noisy, the inclusion of more network-predicted samples might prove more beneficial. 3. The number of Nystrom Samples, denoted as "m," should have an impact on the final output performance. However, the paper lacks a clear discussion on this aspect. In Table 8, there is an option labeled "without tune" with different values of "m." Does this mean that there was no optimization performed on theta and Z? It would be better to include a comparison with optimization and different values of "m" for a comprehensive analysis. 4. Some typos: broken reference in line 41; Eq. 5, for the input point cloud, the logit should be 0.5, on the surface?

Questions

See weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The

Summary

The paper proposes a method for improving shape reconstruction from a set of sparse and noisy unoriented point clouds. As data-driven trained networks can have failures on out-of-distribution examples, this method proposes to take the input point cloud, the predicted surface points, and the features generated by the pre-trained network, and estimate a non-parametric set of kernels to represent the output shape occupancy. Results on shape net, Faust, and scannet show improved reconstruction performance compared to prior deep-network only approaches, and recent work that perform test-time fine-tuning.

Strengths

- [S1] The method outperforms prior work it compares in SoTA deep network shape prediction and reconstructs well thin surfaces. It was nice to see strong generalization results on human shape reconstruction trained only from shapenet data. - [S2] The idea of inducing additional priors and structure in shape prediction is a nice direction to generalization

Weaknesses

- [W1] The related work covers shape representations, optimization-based reconstruction, and feed-forward prediction (either supervised or unsupervised). No context is provided of how this work fits into the field. See Q1 below, the paper needs more motivation and why this particular approach of post-processing using a pre-trained network is a fruitful direction to pursue for improving reconstruction from sparse and noisy point clouds. - [W2] The paper is missing a baseline to classical methods such as PCA-neighbor-based normal estimation + screened-poisson surface reconstruction, using input points and optional pseudo-points from the network prediction. - [W3] No qualitative visuals of failures provided, or discussion of other limitations, such as the assumption that the input point cloud is a contiguous surface and that kernel prior of locally smooth surfaces doesn’t always hold (not just recovering details) - how does the reconstruction perform on an input of several nearby rocks with jagged surfaces and holes? I wish the paper also showed more qualitative examples - only one from scannet and three from Faust are shown. Minor weakness: - Multiple typos, for example section 3.1: (L136 isntance —> instance, L138 contenting —> contending, L148 and —> an, L153 order-invariantly —> pooled to be order invariant, L154 “Without loss of generality we build in this work on the model “). - Writing style - some parts of the text felt extraneous. If the method focuses on post-processing, why devote L136-L147 talking about different input network architecture designs. It seems the proposed method can work with most pretrained-network shape reconstruction methods, as long as it can provide pseudo-labels and features about the 3D space points.

Questions

[Q1] Why is robustifying pre-trained shape reconstruction networks the desired approach to improving shape reconstruction, instead of directly designing a reconstruction method with kernels [1]? What benefit does this alternative direction offers - why post-correct rather than inducing these priors directly in the representation and in the original training. What is the main novelty and contributions over neural kernel fields, and neural kernel surface reconstruction? [1] Neural Kernel Surface Reconstruction CVPR 2023 (and its predecessor neural kernel fields CVPR 2022). The related work cites NKF in “Reconstruction from Point Cloud,” but does not describe how the submitted paper fits in this context. The introduction lacks motivation other than describing the limitations of existing deep-learning based shape reconstruction methods.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

See W3, while they are mentioned, I feel limitations was not adequately addressed with examples. For a visual topic like shape reconstruction, examples would help better illustrate them.

Summary

This paper aims at the task of implicit surface reconstruction from noisy point clouds. It tackles the model generalizability and robustness by introducing a kernel ridge regression as a implicit function that maps a shape feature to occupancy.

Strengths

The proposed method achieve higher accuracy compared to prior surface reconstruction method. It shows certain robustness when applied in real datasets.

Weaknesses

a. What is the motivation behind KRR? How does it help the generalizability of the models? Why it is fitted for surface reconstruction from noisy point clouds? The authors should demonstrate the intuition behind the introduction of KRR, compared to other occupancy decoders. b. From the results on ShapeNet, we see that even for test dataset that have the same density and synthetic/real property as the training dataset, the proposed method still outperforms other methods with a notable margin. Why is that? c. The writing is not very clear, especially how the introduced KRR is integrated in a deep learning network. Is the feature encoder fixed in the proposed method? Is the optimization of KRR conducted for each test sample? If so, the experiments need to compare the test time and accuray simultaneously. Also I have confusion on the discussion from L136-L147. How the intrinsity or extrinsity of point feature could effect the proposed method? Besides, Fig. 1 is not organized well. There too many notations and unclear text color, which does not help understanding. Typos: L42: Wrong reference L45: ; -> , L136: isntance -> instance L148: and MLP -> an MLP

Questions

How the ground truth point clouds of the real dataset (ScanNet v2 and Faust) acquired? What is the difference between ground truth point clouds and noisy point clouds for these real datasets. Are the noise introduced during scanning or added afterwards additionally?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have addressed the limitations.

Summary

This paper proposes an efficient mechanism to address the generalization issues often encountered by generalizable feedforward models trained for implicit shape reconstruction from unoriented point clouds. The solution proposed in this work is to use kernel ridge regression in the network's feature space, tuned by gradient descent. At a high-level, this approach utilizes a pre-trained shape reconstruction network to produce pseudo-labels to provide more supervision. The effectiveness of the proposed method is demonstrated on multiple datasets, including synthetic and real data.

Strengths

1. The overall quality of this paper is high. The sections are well-written and cohesively presented, ensuring a smooth flow of information. 2. The extensive experiments effectively highlight the underfitting and overfitting issues that this research seeks to address. The motivation for the study is convincingly articulated. 3. The decision to implement kernel regression in the feature space of a pre-trained occupancy network is sound and logical. Even though the pre-trained occupancy network is already robust and strong, the approach taken in this paper is understandable given its objective to create a test-time augmentation method.

Weaknesses

1. Comparative Analysis: Given that the proposed method is primarily a test-time optimization technique, it would be beneficial to include a comparison with more test-time augmentation methods. While the authors do mention a test-time fine-tuning baseline in lines 206-207, it might be worth investigating if there are other advanced test-time optimization techniques, specifically ones that could directly apply to Poco features. 2. Demonstrating Generalizability: The results showcased in this paper are primarily based on the Poco network, which already significantly outperforms other baseline methods as demonstrated in Table 1. Although I believe this method could be applied to other feature extractors, it would strengthen the paper to present more numerical results to support this.

Questions

1. What does the k_theta in L184 represent?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The paper has discussed its limitations.

Summary

This paper aims to reconstruct smooth and high-quality surface from noisy point cloud. While previous works like Poisson surface reconstruction can extract surface from point cloud, but it's sensitive to input noise. The existing learning-based methods are able to handle noisy point cloud, but often suffer from generalizing to other dataset, real-world scenes, and large scale point cloud. The authors propose to leverage pretrained learning-based occupancy network (POCO) for extracting point cloud features and generate pseudo label for new points. The Kernel Ridge Regression (KRR) and the Nystrom samples are jointly optimized with gradient descent, representing the final shapes from the input point cloud. The experiments show that proposed method achieve better evaluation metrics (CD, IoU, NC, FS) on synthetic ShapeNet, real-world dataset ScanNet v2. The qualitative results and ablation study are conducted to verify the design choices.

Strengths

- The quantitative results in Table 1-6 demonstrate that proposed method achieve better evaluation metric, including Chamfer distance, IoU, normal consistency, and F-sore. These metrics indicate the reconstructed meshes have superior quality. - The qualitative results in the figures show that the reconstructed surface is smoother and more complete, in contrast to the baseline methods. - The proposed method can generalize to synthetic and real-world large scale input point cloud.

Weaknesses

- Formulating implicit surface with kernel-based method is not a new idea. The fundamental difference between proposed method and previous works "Neural Kernel surface reconstruction", "Neural Fields as Learnable Kernels for 3D Reconstruction" is not described in the paper, and the comparison is also missing in the experiments section, making it difficult to evaluate the technical contribution. - The propose method relies on pretrained POCO to generate point features and pseudo points label. The authors argue that the learning-based methods suffer from generalizing to out-of-distribution data. As a result, the proposed method is likely to fail for the data that POCO cannot produce good enough initialization (e.g. incorrect occupancy prediction). As a result, the capability of generalization of proposed method does not outperform previous learning-based method significantly. - Per-scene optimization is required to get KRR hyper-parameters and Nystrom sample points. While the optimization doesn't take much time, the comparison with baseline methods might be unfair since most of them only need single forward pass. In addition, according to the quantitative results in the tables, the number only improve slightly from POCO baseline. - The weakness of previous works and difference of proposed method is not clear in the related works section. To be more specific, in the subsection "Unsupervised Implicit Neural Reconstruction", various previous works are mentioned along with some technical details, but the description about the their common limitation is missing, the only text that describe this group of works is "All of the aforementioned work benefits from efficient gradient computation through back-propagation in the neural network." which does not contain much information.

Questions

- What's the fundamental difference between the proposed method and previous kernel-based neural surface reconstruction (see the references in the limitation section)? - How could the optimization recover the shape surface from point cloud if the POCO backbone is not able to produce reasonable prediction? - Is there any intuition behind the design choice where the authors decide to represent the surface as occupancy instead of signed distance function? ### Other comments - Missing section reference in line 42 - typo in line 148 - Missing bracket in eq. 12

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

There's no potential negative societal impact of this paper.

Summary of changes

We thank the reviewers for their positive reception of our work and appreciation of our rebuttal. Thanks to their valuable feedback from both the reviews and the discussion we improved the quality of our manuscript through the changes summarised below. More details on these elements can be found in the rebuttal. **Introduction**: We clarified why generalizable shape networks can fail (underfitting the input). We clarified motivation of improving on pretrained shape networks (training such large models can be resource intensive, adaptation is less costly and yields considerable improvement), doing so with a learnable N-KRR (an alternative to unstable network weight tuning that offers a shape specific expressiveness/regularization tradeoff), and using a Gaussian kernel (universality properties of the associated RKHS, adaptation to preexisting features). We reorganized and improved writing. **Related work**: We clarified relation and differences from [76,78] (although both our inputs and strategies differ, c.f. rebuttal), and introduced concurrent work NKRS CVPR23 (although our strategies differ). We added some missing weaknesses of prior work. We reorganized and improved writing. **Method**: We made Fig. 1 more intuitive, and explained its color coding further. We reorganized and improved writing to be clearer. **Results**: We added justification of choice of number of augmented points, comparison to SPSR, the generalizability result (using Conv [55] as a baseline for our method), ablation of number of inducing points “m” under optimization. We also added more visual results. **Limitations**: We added a discussion of the effect of high “m” values on overfitting, a discussion and visual results of behavior under baseline failures, with the hypothesis and evidence of our improvement.