Summary
The manuscript presents a differentiable pipeline for generating watertight manifold meshes from multi-view images observation. The NeuManifold leverages TenorRF [Chen et al, 2022a] to recover a density-based representation. The density is then converted to SDF and later meshes using differentiable marching cubes presented in FlexiCubes [Shen et al, 2023]. The final optimization is baed on nvdiffrast [Laine et al, 2020] is done for improving details. The performance of NeuManifold is compared with prior work on the public available NeRF-Synthetic dataset.
Strengths
- The paper is well-written and clear to follow. The figures are illustrative to showcase the pipelines and performance variations.
- The present idea is intuitively sound (except for the first component), with each stage performing meaningful tasks.
- The problem is of interest to the research community, as image-based watertight reconstruction can be widely applicable in many industries.
Weaknesses
- Theoretical Limitations
- The suggested approach is asserted to possess the capability to restore watertight manifold meshes. Nevertheless, these meshes are directly generated from density data with marching cubes, and as a result, there could be arbitrary noisy surfaces extracted, unlike more principled representations such as SDF. The absence of a theoretical framework to ensure a principle density formulation towards watertight mesh contradicts the claimed contributions in the manuscript.
- Lack of Innovation:
- Although the overall workflow appears logically coherent, the suggested method primarily amalgamates various pre-existing approaches. The rendering networks draw inspiration from TensorRF [Chen et al, 2022a], the notion of differentiable marching cubes originates from FlexiCubes [Shen et al, 2023], and the concept of differentiable rasterization is credited to nvdiffrast [Laine et al, 2020].
- Inadequate Experiment Results:
- The manuscript asserts that "SDF-based techniques often suffer from reduced visual fidelity and the loss of high-frequency geometric details" (Page 4, Section 3.3). However, the paper overlooks recent breakthroughs in SDF-based neural surface reconstruction algorithms capable of generating watertight, high-fidelity meshes from images. Numerous recent advancements, exemplified by Neuralangelo [Li et al, 2023], NeuS2 [Wang et al, 2023], and HF-NeuS [Wang et al, 2022], have emerged in this field. Without comparing against such approaches, it is difficult to understand how much progress NeuManifold has made. Furthermore, building NeuManifold on existing SDF-based approaches seems to be a more logical choice.
- Moreover, the current results are only constrained to synthetic scenes using NeRF-Synthetic. It will be interesting to see results from real-world captures.
References:
Li, Zhaoshuo, Thomas Müller, Alex Evans, Russell H. Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. "Neuralangelo: High-Fidelity Neural Surface Reconstruction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8456-8465. 2023.
Wang, Yiming, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. "Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3295-3306. 2023.
Wang, Yiqun, Ivan Skorokhodov, and Peter Wonka. "Hf-neus: Improved surface reconstruction using high-frequency details." Advances in Neural Information Processing Systems 35 (2022): 1966-1978.
Questions
- What are the theoretical novelties of the framework such that it is more suitable to the ICLR audience?
- Can authors provide comparisons against the prior works using SDF-based surface reconstruction approaches, such as those listed above?
- How well does the pipeline work if the NeRF is based on recent SDF-based surface reconstruction methods?
- How well does the approach work for real-world captures?
- The manuscript claims in Figure 5 that it is safe to assume axis alignment between marching cubes coordinates and real-world objects. Without explicit alignment, this may not be the case. Can the authors provide more details on this?
- Figure 3 compares marching cubes/tetrahedral results with and without “exp()” on SDF fields. Why is this important given SDF is already meaningful for surface extraction?
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.