Inferring Neural Signed Distance Functions by Overfitting on Single Noisy Point Clouds through Finetuning Data-Driven based Priors

Summary

The paper presents a method for learning neural signed distance functions (SDFs) from noisy point clouds. This approach integrates the advantages of data-driven and overfitting-based methods to enhance generalization, accuracy, and inference speed. They employ statistical reasoning within local regions to refine the estimation of SDFs without requiring clean point clouds or signed distance supervision.

Strengths

1. This paper successfully combines data-driven and overfitting-based approaches, leveraging their strengths while mitigating their weaknesses. 2. This paper shows that the method enhances robustness against noise, a common real-world challenge. They effectively use local statistical reasoning to adjust the neural SDFs, improving accuracy in surface reconstruction and denoising tasks, as evidenced by the benchmark results.

Weaknesses

1. There are a lot of typos in the manuscript, e.g., In the Abstract and Introduction, "stat-of-the-art" should be "state-of-the-art"; Line 57: "Learning implicit functions have achieved" should be "Learning implicit functions has achieved"; Line 39: "overfits an neural network" should be " a neural network", "Querys", and so on. In the Abstract, the "prompt" might be "promote". It would be best to check out all the typos. 2. Dependency on initial conditions: The performance might heavily rely on the quality of the initial data-driven priors, which might limit the method's effectiveness when such priors are not well-tuned or applicable. 3. The paper does not extensively discuss the computational demands or scalability when applied to larger datasets (e.g., nuScenes and Waymo) or more complex scenarios.

Questions

1. How does the method perform under limited computational resources? When scaling to larger datasets, is there a significant trade-off between accuracy and computational efficiency? 2. Can the method maintain its performance across different types of noise or corruption that were not part of the training set? 3. How does the method perform when such priors are unavailable, or how sensitive is it to the initial conditions set by these priors?

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

See the "Questions" and "Weaknesses" sections above.

Summary

The presented work tackles the problem of reconstructing a shape from a noisy point cloud (PC) into an implicit representation, using the signed distance function (SDF). Recent methods are categorised into i) data-driven approaches that learn a shape prior with a dataset of training shapes, with poor generalization but generally faster, or ii) approaches that overfit on a single PC at a time, with better generalization but slow convergence. The paper proposes a middle ground: finetuning a pre-trained shape prior model to a single PC, using a novel approach that performs "local statistical reasoning": supervising local parts of the shape with multiple sampled noisy PC. The method is experimentally validated against multiple baselines, either data-driven or overfitting-based, and on multiple datasets showing better reconstruction accuracy at lower convergence times. A final ablation study shows that the prior and locality of the loss play a role in the method's performance.

Strengths

1. Experiments are extensive: the PC reconstruction is evaluated against multiple baselines (both data-driven or overfitting-based) on various datasets (ShapeNet, ABC, FAMOUS, SRB, and D-FAUST). In nearly all cases and presented metrics, the proposed approach is the best. 2. I found the locality approach to solving the problem interesting; supervising parts of the shape at a time instead of the full one. 3. Using a shape prior that is unrelated to the target object (such as ShapeNet prior for FAMOUS reconstruction) also works. This suggests that this approach is applicable even if the target shape is from an unknown category (thus helps for generalization). 4. The method is fairly simple and clear, which should make it easily usable.

Weaknesses

1. The main pipeline is fairly simple (fine-tuning a pre-trained model). I believe the locality of the "statistical reasoning" is perhaps the main novelty, and the ablation shows that it improves accuracy and convergence time (§ at L.342), but it is not discussed or explained why using a local approach instead of a global one improves convergence and accuracy. 2. The writing could be improved. More specifically, the experimental results on the different datasets all make more or less the same points and so could be condensed into something more compact. This can free up some space to expand other parts and increase their clarity, e.g., the related works and the "Neural Signed Distance Function." paragraph (L.101). 3. Throughout the work, the noise is assumed (additive) Gaussian. While this is not a weakness in itself, this still is an assumption that should be explained and made clear: 1. Why is a Gaussian noise model good? Are Lidar data, for instance, usually with gaussian noise and what real use-cases does it apply to? Generally speaking, the motivation of the work could be expanded beyond "reconstructing a PC to an SDF". 2. "Statistically, the expectation of the zero-level set should have the minimum distance to all the noisy point splitting." (L.131-132) This is true assuming Gaussian noise with expectation 0. With other noise types the loss may not perform as well (e.g., "Text removal" in Noise2Noise [1]). [1] Lehtinen et al., Noise2Noise: Learning Image Restoration without Clean Data, ICML 2018.

Questions

1. What is the reasoning behind using the locality in the "statistical reasoning" and its better performance? 2. For the ShapeNet dataset, is it per-category and then averaged or trained on multiple categories combined? (and which ones?) Same questions regarding the SRB/FAMOUS experiments with ShapeNet prior. 3. What are the PC sizes? It's hard to appreciate the other parameters without this info. What are the datasets sizes in number of training and test shapes? 4. How does the prior help if it apparently does not need to be from the same dataset as the test shapes (e.g., SRB/FAMOUS experiments)? Is it actually pre-training on any valid SDFs that helps? If so, how do you think it compares to the simple SAL [2] geometric initialization? 5. Why are the metrics (CD-L2/L1, NC, F1-score) not used consistently through the datasets and results? [2] Atzmon and Lipman, SAL: Sign Agnostic Learning of Shapes from Raw Data, CVPR 2020.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitations are briefly mentioned in the appendix regarding noise level. What kind of noise level is considered too big? It might also be worth commenting on the noise types (see Weakness 3.), is the method working only for Gaussian noises?

Summary

The authors propose an implicit surface reconstruction method from point clouds that uses a learned prior to initialize the optimization of a neural SDF. First, a neural SDF generator based on DeepSDF [66] is trained on a dataset of shapes to learn a prior over shapes. Given a point cloud, both the shape code used as input for the generator, as well as the parameters of the neural SDF are then optimized to fit the point cloud, using an approach based on Noise to Noise Mapping (N2NM) [50], but focusing on local patches instead of the full global point cloud. The authors show that this approach performs better than N2NM as well as several state-of-the-art methods in an extensive evaluation on several datasets.

Strengths

- Using a learned prior as initialization for an optimization-based surface reconstruction method like N2NM has not been done before as far as I can tell. - Using local patches also seems like a (smaller) contribution to me (the authors show that this contributes to the performance, but it is not very clear to me at this point why it improves performance significantly - this could be improved). - Results show consistently better reconstruction accuracy than the state-of-the-art. - The evaluation is extensive and mostly well-done, and convincingly shows and advantage over the state-of-the-art.

Weaknesses

- While the idea of using a learned prior as initialization of an optimization based method has not been done for surface reconstruction as far as I can tell, it is not very surprising that this can be done and is also relatively straight-forward to do. Both stages are essentially the same as existing work [66] and [50], with some deviation from [50] in using local patches. So all-in-all, the technical contribution does not seem huge, although possibly still large enough for acceptance. - While local patches do seem to improve performance, it is not fully clear why this is the case, and the exact details of using local patches are missing some details in the exposition, making it unclear exactly how they are implemented. - A few recent methods are missing both from the related work and the comparisons (see below). - The exposition could use some additional details and clarifications in some parts (see below). Most of the weaknesses, apart from the first one, can be addressed with text changes to some extent. Given the good performance of the method and the extensive evaluation, I lean towards acceptance. Details: - The field of surface reconstruction from point clouds is quite vast, so the authors missed a few works: - Neural-Singular-Hessian: Implicit Neural Representation of Unoriented Point Clouds by Enforcing Singular Hessian, Wang et al., TOG 2023 - Iterative Poisson Surface Reconstruction (iPSR) for Unoriented Points, Hou et al., TOG 2022 - 3DShape2Vecset: A 3D Shape Representation for Neural Fields and Generative Diffusion Models, Zhang et al., TOG 2023 - Geoudf: Surface reconstruction from 3d point clouds via geometry-guided distance representation, Ren et al., TOG 2023 - Given good normals, surface reconstruction becomes much easier - the normal computation could be followed by Poisson reconstruction based on the normals for example. Therefore papers to compute oriented normals are relevant, such as: - Orienting Point Clouds with Dipole Propagation, Metzer et al., TOG 2021 - Globally Consistent Normal Orientation for Point Clouds by Regularizing the Winding-Number Field, Xu et al., TOG 2023 - SHS-Net: Learning Signed Hyper Surfaces for Oriented Normal Estimation of Point Clouds, Li et al., CVPR 2023 - The exposition could use some additional details and clarifications: - The approach for using local patches is missing some details: How many patches are used during optimization? In which order are patches evaluated? One after the other? Or is a random patch chosen in each iteration? This should be clarified. - The difference of the second stage to [50] could use some more discussion. The authors describe the application of [50] to local patches as main difference (as shown in Eq. 3). But since the expectancy in Eq. 3 is taken over random samples near random patches, how is this different than taking the expectancy over random samples near the whole shape (as in [50])? This could use some discussion. (Or is the SDF actually optimized separately per patch? From reading I did not get this impression.) - [50] also uses a geometric consistency constraint as regularization, although such a regularization is not described by the authors. If it is not used, a discussion why the authors removed it might be useful. - c seems to be initialized to a fixed constant vector: the average of all embeddings in the training set, while the original DeepSDF paper uses a c randomly sampled from a Gaussian. Why did the authors use a different initialization for c here? Did the average empirically perform better? This could be discussed. - In the paragraph starting at line 129, It should be stated clearly that the method described here is the method proposed in [50] (although evaluated at random samples near random local patches, rather than random samples near the whole shape). - The evaluation is quite extensive, but would benefit from a few improvements: - In the evaluation, it would be good to hear more about how the time used to fine-tune each optimization-based method was chosen. Was each result optimized until convergence? Or otherwise, how was the number of iterations/optimization time chosen? - It would be good to add DeepSDF to the comparison, as it is the closest to the initialization used for the optimization stage, and would show how much the fine-tuning improves accuracy. - In the ablation, using a large local region seems worse than using a global approach without local regions. This should be discussed. - References [52] and [53] are duplicates

Questions

- A clarification of the unclear details regarding local patches would help. - A clarification of how timings for methods were chosen would help as well.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have not discussed clear limitations unfortunately.

Summary

The paper proposes 3D shape reconstruction given a noisy point cloud using a test-time optimization approach. The paper proposes a new loss function that can work directly on noisy point clouds without the need of ground truth normals or SDF. The approach involves learning a network (DeepSDF) to predict SDFs and later fine-tune the network on a single shape during test. The learned priors reduce the test-time fitting and leads to better reconstruction results.

Strengths

1. The paper proposes a simple and easy to implement approach to 3D reconstruction that works on a bit noisy point clouds. 2. The better quality results and fast inference has a broad significance in 3D reconstruction.

Weaknesses

1. In the introduction, the test-time optimization is hailed as a novel thing, however previous methods (including DeepSDF) has already proposed it. 2. The main contribution is the loss function defined using "pulling" operation, which forces the surface defined by the projected points close to the input points be similar to the input point clouds. However, why this loss function should work in a noisy point cloud setting is not well motivated and well explained. 3. The writing of the method section can be significantly improved. For example, the symbol $f$ seems to represent both the neural network that predicts the signed distance field and the field it self. More about it in the Questions section. 4. The use of the phrase "statistical reasoning"is quite vague. The phrase is used throughout the manuscript but does not convey anything meaningful. 5. It is not clear whether the baselines in the experiment section are also trained on noisy data to make them robust. 6. Comparison with Points2Surf [21] method is missing. I think it work quite well on shapes in Figure 4. 7. What does "extreme noise" mean in the Table 13? What does it translate to in percentage of the scale of the scene. 8. How do you achieve 100% sparsity in Table 14? Does it imply no points? 9. Noise level of 0.5% is quite low to claim that the method is robust.

Questions

1. Please improve the notations in the method section. The vectors should be bold. Use different symbols for parameterized function and output. 2. The motivation of the loss function is not clear. Perhaps a diagram would enhance the understanding for a reader.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

1. Limitations regarding pose variations and incomplete data (holes) are not discussed.

We value the opportunity to discuss with you

Dear reviewers, Thanks for your valuable comments. We will be happy to take any questions. Please let us know if our rebuttal address your concerns. Thanks, The authors