RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency

Summary

This paper proposes MucRays, which imposes ray-based neural functions with multi-view geometry consistency. This framework contains three-parts: ray-surface distance field, dual-ray visibility classifier, and multi-view consistency optimization strategy. Quantitative results of MucRays surpass the existing coordinate- or ray- based networks for synthetic or real-world scene modeling. This ray-based method achieves 1000x faster for depth image inference.

Strengths

1. The dual-ray visibility classifier is interesting. It is well-motivated and acts as the foundation of the multi-view consistency optimization. 2. The quantitative results are good. The high performance of MucRays on DAE metric proves its ability to render more accurate depth images.

Weaknesses

1. The qualitative results does not satisfy me. It seems that, MucRays has difficulty in representing thin structures. For example, in the Reception scene in Figure 4 (Appendix), the arm of the desk lamp is missing in both distance map and mesh. I would like to see a discussion about such phenomenon. 2. I think NeuS (NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction) is a better baseline than DeepSDF because it could use both depth and color for supervision. 3. I assume that this paper focused on better geometry modeling. However, it seems that the geometric result (mesh result) on Lego (Figure 5) is over smoothed compared with DS-NeRF or NDF. Is it caused by TSDF or the nature of ray distance representation itself?

Questions

Main questions are in Weakness part. More detailed comments: The authors should use consistent abbreviation. For example, L. 80 use UDF for paper [13], while the remaining contents use NDF for [13].

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes.

Summary

This paper presents a new strategy to design ray-based neural representation of 3D shapes. Ray-based approaches towards shape representation is a recently emerging idea that bypasses extensive point-based evaluation required by conventional methods like signed distance function. A key missing component in existing ray-based neural representations is the lack of multi-view consistency. This paper proposes training a dual-ray visibility classifier and using that network's output to guide the training of the network for ray-distance prediction. The method is extensively evaluated on synthetic and real-world datasets, showing its superiority over previous ray-based methods in predicting ray hit points from novel views. Moreover, this paper also includes evaluations of color prediction of this ray-based method, showing it achieving comparable performance to previous efficient representations.

Strengths

- Addresses a key issue of multi-view inconsistency in current ray-based scene representations. - Clear writing and figure presentation. - Extensive evaluation showing the efficacy of the proposed strategy, outperforming previous baselines without slowing down inference

Weaknesses

- The proposed method operates under the setting where depth maps are available for all views, which is okay, but the paper would be more complete if it discusses how bad the results would become if depth is unknown, which is actually the setting of LFN. - Not totally clear how the baselines are implemented. For example, when using LFN and PRIF, the supplementary material mentions that it uses the same architecture as the official setup. However, it is not clear whether the LFN/PRIF results are obtained still with the proposed classifier. At least based on the writing, it does not eliminate the ambiguity. - The proposed method also adopts a different ray parametrization than LFN and PRIF. It is not clear what the reason is behind this design choice. - The proposed method is implemented as a 13-layer SIREN with 1024 hidden channels, which is different than previous baselines. The paper does not include any analysis on the impact of varying layer depth and hidden channels. - In L123, the paper says the network predictions "must satisfy a transformation equation", without defining what this equation is. - Related work misses any discussion on space carving / visual hull (e.g., [Matusik 2000], [Kutulakos 2000]), which are important early concepts behind the idea of visibility checking proposed in this work. Related work also misses some recent papers on light field / ray-based representation (e.g. HyperReel [CVPR 2023], SRT [CVPR 2022], SIGNET [ICCV 2021]). - L237 typo: "RTX 30390 GPU"

Questions

- In Table 5, the performance drops significantly after removing the classifier, and it actually becomes much worse than the various baselines in Table 2. This is confusing and relates to the above-mentioned weakness about "Not totally clear how the baselines are implemented". Do the baseline methods included in Table 2, specifically LFN and PRIF, involve a classifier? That would totally change how to correctly interpret the results in Table 5. - If the baseline methods do not involve a classifier, and they are only just about a different ray parametrization with slightly smaller network, then the natural question becomes: how does LFN/PRIF + classifier perform? - In general, how does the ray parametrization affect the performance? It appears that it would have minor impact compared to the role of classifier, but the paper does not provide enough information to draw any conclusion.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

The paper proposed a ray-based neural rendering method that is able to achieve good reconstruction quality from depth maps or RGBD inputs using only one network evaluation per pixel during evaluation. The method requires two networks: a ray-surface distance network, and a dual-ray visibility classifier. In the first stage, the visibility classifier is trained on the multi-view depth maps. In the second stage, the ray-surface distance network can be trained with both ground truth rays from the depth maps, and random sampled novel rays. the visibilities of the random rays are determined by te visibility classifier, and only the distance of the visible rays are supervised. The proposed approach is extremely fast, and achieves competitive quality on three RGBD datasets.

Strengths

* The proposed two-stage method is novel. * The proposed method is extremely fast during evaluation -- only one network evaluation is needed per ray. * The method achieves competitive performance on both synthetic and real datasets. * The paper provided useful insights on the derivation of surface normals from the proposed ray distance field.

Weaknesses

* The method requires ground truth depth information for training, which is not the case for light field works such as [57]. This can significantly limit its use cases. * There lacked comparisons with some good performing light field methods such as [57]. * The method relies on the visibility classifier to generalize to novel rays, which can potentially be unreliable. * Ther performance of the proposed method is lacking, especially according to the results in the appendix. * It might help understand the effect of visibility network better if there are visualizations of the visibility network predictions.

Questions

There appear to be some fixed-pattern noise artifacts in the demo video. I wonder what is the cause?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations and broader impacts of the works are adequately discussed in the paper.

Summary

This paper proposes a framework MucRays for 3D shape representation. Specifically, the authors formulate 3D shapes as ray-based neural functions and incorporate multi-view geometry consistency to improve the performance. For the learning of multi-view geometry consistency, an auxiliary network is introduced to classify the mutual visibility of two sampled rays. Experiments are conducted on various datasets, showing good performance.

Strengths

1. The paper is easy to follow and understand. 2. The authors propose an effective framework for 3D shape representation, showing good performance in various datasets.

Weaknesses

1. This paper only compares rendering time, but training/optimization time is not compared. 2. The ablation study is not convincing. The authors should compare with and without postprocessing (Section 3.2). 3. Table 5, the performance of "w/o classifier" is quite worse than the full model, and the error is very high. It would be better to discuss some possible reasons. 4. 2-stage training is not elegant. Is that possible to train these two networks simultaneously? 5. Line 57, "ray-surface distance field" representation is introduced by previous works and cannot be summarized as one contribution of this work.

Questions

See weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Negative impacts are mentioned in the paper.

Reply

The rebuttal has solved my concerns and I maintain my initial rating.

Thanks for the rebuttal

I would like to thank the authors for the clarifications, as well as the extra experiments and figures. I decide to keep my rating of weak accept. Although the paper has proposed an interesting way to model 3D geometry, its drawbacks are also quite significant: it requires depth maps to train, and the quality does not stand out among baselines.

Waiting for Discussion

Dear reviwer fezd, Thank you again for your initial valuable feedback on our manuscript. While we understand that your time is very demanding, we are still waiting for your thoughts on our rebuttal materials (summarized in the "Author Rebuttal by Authors" section). Regarding all your concerns including the training speed, more ablation studies, single-stage training, etc., we believe they are all clearly addressed above. We would greatly appreciate any additional comments you could provide. Your time and consideration are highly valued. Regards, Authors