Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors

Summary

This paper tackles the problem of 3D scene reconstruction from posed RGB-D images by learning an implicit occupancy function. Unlike previous methods [58, 3, 65, Ref_DS] that directly use the depth values (obtained either from depth images, SfM pipelines, or from depth prediction networks) of individual rays (i.e. pixels) as supervisory signal leading to learning better scene geometry, faster convergence [Ref_DS, Ref_IN] and inference [Ref_IN, 58], this method proposes to first fuse the depth frames into a TSDF representation using some off-the-shelf method and then query the TSDF for obtaining the occupancy value corresponding to a particular point on a sampled ray for supervision.In other words, the off-the-shelf obtained TSDF of the scene is used as a scene prior. The motivation for relying on TSDF representation of the scene is that the depth images obtained from RGBD sensors suffer from missing depth values (or holes) which makes it challenging to supervise 3D point samples corresponding to those rays. The primary contribution of this paper is to learn an attention function that, for a 3d point sample, estimates the weights for the linear combination of occupancy values which are predicted from the neural network and occupancy values obtained from the TSDF scene prior (as in, final_occupancy = w1*predicted_occupancy + w2*TSDF_obtained_occupancy_prior). In this sense, the TSDF prior becomes an attentive depth fusion prior. The network is supervised using volumetric rendering of depth and colour as proposed in [31]. Just like previous methods (e.g. [48, 58]), this method too relies on a hybrid representation: three discrete feature grids are learned, two for low and high-resolution representation of geometry and one for colour, and the tri-linearly interpolated feature value for a corresponding 3D point is decoded by corresponding MLP decoders to obtain occupancy and colour for that 3D point. The truncated nature of TSDF (it belongs to (-1 , 1) ) is also exploited to inform when to use low-resolution grid features to get occupancy and when to rely on both low and high-resolution grid features to predict occupancy and fuse it (using learned weights) with the TSDF informed occupancy prior; it is clear that points which have TSDF values under the range of 1 or -1 are near surfaces and hence higher resolution feature grids are necessary and in addition, TSDF will explain such points the best. The method is also applicable for SLAM settings where RGB-D frames are obtained incrementally in which the camera extrinsics, i.e. poses, are also optimized with the network parameters. [Ref_IN] Kim, Mijeong, Seonguk Seo, and Bohyung Han. "Infonerf: Ray entropy minimization for few-shot neural volume rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. [Ref_DS] Deng, Kangle, et al. "Depth-supervised nerf: Fewer views and faster training for free." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Strengths

1. The paper, barring a few places, is well written. 2. This paper has identified the limitation in supervising neural networks that learn implicit functions for dense 3D scene reconstruction using depth images which is difficult in supervising predictions for points whose depth values are not available as they belong to holes in the depth images. 3. Authors have put in a reasonable effort to analyse the proposed method with good qualitative and quantitative analysis.

Weaknesses

1. The paper lacks any substantial contribution Explanation: a. It relies on hybrid representation for learning implicit functions for dense 3D reconstruction which has been very well explored before in the same context ([48], [58], [19]) b. The use of pre-trained MLP decoders for predicting occupancy from trilinearly interpolated feature vectors from multi-resolution grids has been inspired by NICE-SLAM[65]. c. The TSDF for the scene is obtained by fusing the posed depth images using the off-the-shelf method. d. The only contribution of this paper is towards learning an attention function that predicts the weights which are used to linearly fuse the predicted occupancies and occupancy obtained from a TSDF prior. 2. Given the way the attention function is learned which is the primary contribution of this work, it almost looks unnecessary : Explanation: As explained in the method section and also schematically shown in the pipeline figure (fig 1), the attention function takes the sum of the predicted low and high-resolution grid occupancies and occupancy obtained by TSDF prior (normalized between 0 and 1) to predict the weights for the linear combination of the two (eq. 2, eq 3., figure 1.). This means the attention function only chooses to learn the weights (alpha and beta, eq. 2, 3) by looking at only two floating point numbers which are between 0 and 1 based on errors in RGB and depth renderings. The RGB and Depth renderings are obtained from the final linear combination of the two occupancies (eq. 2, 3) and their quality would be inversely proportional to the rendering errors. I don't see what else the attention function can learn to pay attention to other than which one of the inputs is larger. In which case, how would it be any different from simply using the softmax operator to combine the two occupancies instead of learning two weights? Had the attention function taken any feature from grids to reason about how well the network can represent the scene , it could make better decision as which is more important, predicted occupancy or TSDF prior occupancy. Accordingly, it would predict weights for the fusion of the two. 3. The method is affected by the very problem it is trying to solve: Explanation: The paper claims that missing depth values are a problem when using them raw for learning implicit functions for scene reconstruction. The fused TSDF prior would help mitigate that. However, from Figure 2 it is clear that TSDF prior could also have holes where the network has to rely on its own learned occupancy. This shows that the idea of using TSDF is itself affected by the very problem it is trying to mitigate. 4. The paper also proposes that the missing depth due to occlusion leads to difficulty while supervising using depth images. However, when posed RGB-D frames of the scene are available, the points that are occluded in one frame would be visible in some other frames anyway. Consider, for example, the classic NeRF [27] paper or [Ref_DS], the novel view renderings are not affected by occlusions in different frames as the occluded ones are anyway visible in other views. 5. The proposed method is very similar to [19] in the sense that both rely on fused TSDF of the scene then it would be interesting to see how the proposed method fairs with [19]. The qualitative results of [19] look quite similar to the ones shown in this paper which motivates me to also consider a quantitative comparison between the two. 6. The paper points the reviewer to refer to supplemental material for more results. But there is no supplemental material pdf for reference (the supplemental material also has the paper in it). [Ref_IN] Kim, Mijeong, Seonguk Seo, and Bohyung Han. "Infonerf: Ray entropy minimization for few-shot neural volume rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. [Ref_DS] Deng, Kangle, et al. "Depth-supervised nerf: Fewer views and faster training for free." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Questions

1. How is the proposed attentive depth fusion prior different from simply using a softmax operation over the predicted occupancies and the occupancy obtained from TSDF prior? This question is motivated from the fact that the attention function only takes the two occupancies as input . 2. How is the error in TSDF which is accumulated due to progressive fusion of depth images using their recent estimate of pose handled? Detail question: In a SLAM setting where we do not have all the RGB-D images beforehand, the TSDF is performed by fusing the scene incrementally using the estimated poses. However, pose estimates from any front-end would have some degree of error. This error would propagate to the TSDF scene prior which is used to construct the final occupancy. The network however is optimized along with camera poses but the TSDF does not seem to be updated based on the optimized camera poses. If, however, it is updated, would it not mean that at every mapping phase, we will have to fuse the entire TSDF again? This would again mean that we will have to keep appropriate number of depth frames for this. However this method, like vMAP[17], iMAP[41], stores keyframes, whose main purpose is for replay while learning incrementally to tackle the catastrophic forgetting issue. No reference to this is provided in the paper. 3. How would the result look like if instead of a linear combination of the predicted occupancy and occupancy from TSDF prior, we use the one with the maximum weight? Could the results indicate that the attention function is redundant? This would also demonstrate if there is any merit in the idea of using TSDF as a prior for composing the occupancies or learning to predict TSDF from TSDF prior is a better idea; [19] has three stages of training, the first is devoted to this.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

The authors have not discussed any limitations. However, the following are the two strong limitations of this work: 1. One strong limitation of this work is that it would require the availability of TSDF prior even in test time. So what is the point of learning implicitly? The method is also capable of optimizing the camera poses and hence the resulting occupancy would be better (as also shown in the results). But then how is it different from running a classic bundle adjustment and then fusing the optimized depth values into a more accurate TSDF using optimized camera poses? Whereas methods like MonoSDF[58], FastSURF[19] do not need fused TSDF in the test time and the results are quite comparable if not better. 2. Ideally, the attention function could be also provided with the features from the grids so that the function could learn to decide upon the fusion weights also based on the features which are also being optimized. In this process, the learnt features would be considered when deciding the weights required for fusing the occupancies. This would help the network to actually reason about the fusion based on the geometric understanding learned so far.

Summary

This paper proposed a simple yet effective approach of neural implicit surface reconstruction from RGB-D seuqences that leverages the reconstructed TSDF grids as a prior which effectively improve the reconstruction quality of fitting a neaural implicit surface from multi-view RGB-D images directly. The idea of using fused TSDF grids as a geometric prior is novel and makes sense as it comes for free from RGB-D sequences. The proposed attention mechnism of fusing the results from TSDF prior and neural implicit function is also novel and effective through experimental results. The authors have also demonstarted the effectiveness of their proposed methods and each of its components through extensive experiments, ablations and analysis. Except from some very minor issues, this is a good paper. For these reasons, I would reccomend "Weak Accept" at this stage.

Strengths

1. The idea of using fused TSDF grids as a prior is novel. Normally, traditional TSDF fusion could achieve better accuracy in geometry reconstruction but falls short of filling the holes in occluded or unobserved regions. Neual implicit methods, on the other hand, is better at hole filling at the cost of slightly less accurate surfaces. Therefore, using TSDF fusion as a prior and combining these two methods is very natual and has very clear motivation. Also as the authors showed, fused TSDF grid already encoded all historical frames and is occlusion-aware, which could also bring benefits to directly fitting to multi-view RGBD images. Moreover, this prior also comes with no extra cost. The proposed learnable attention module to fuse the output from two sources is also a "simple-yet-effective" solution. 2. Overall, the quantitative experiments are conducted extensively and are sufficient to show the effectiveness of the proposed methods. Quantitative results show the proposed method achieves SOTA reconstruction and tracking results compared to previous RGB-D neural implicit SLAM/surface reconstruction methods. Qualitative results also look promising and accompanied with detailed and clear analysis. Ablation studies are also well conducted and justified the effectiveness of each module and design choices. 3. The paper is very well written and easy to follow. The introduction and related work sections are very clear and show clear motivation of the proposed method. 4. The method could run both in an online SLAM manner and under batched offline setting, which increases the system's practical value.

Weaknesses

1. Overall, the comparisons with RGB-D baselines seems convincing, but some of the experiments in Sec. 4.2 are comparing against baseline methods that rely on RGB-only input. For example in Tab. 2, 4 and Fig. 5. These are not completely fair comparisons as the proposed methods take measured depth directly. Or did you change some settings in the experiments, like supervising the proposed method with predicted depth instead of measured depth? 2. It would be good to also include performance analysis, such as memory and run-time comparison to previous methods, especially for the SLAM setting.

Questions

Some minor questions: 1. What's the resolution and voxel size used in the feature grids and TSDF grids? 2. Regarding the learnable attention module $f_a(\cdot)$, seems that it is only conditioned on the two occupancy predictions? Have you tried with more conditions? 3. When evaluating reconstruction quality on ScanNet (Tab. 4), what mesh culling strategy did you use? Also is it consistent across all the methods? 4. In the ablation studies on depth fusion (Tab. 8), why would there be difference between GT All and GT? If I understand correctly, the difference is just that GT All operates in a batched manner while GT is online. But this shouldn't cause difference when you run TSDF-fusion?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes

Summary

The paper proposes a pipeline for estimating scene geometry with TSDF represented with neural implicit function, based on multi-view RGBD inputs. The main novelty is a fusion mechanism which utilized fused depth geometry as prior, and fuses geometry prior with estimated geometry using attention-based weighting. The weighting also considers bandwidth of the TSDF representation, allowing for multi-resolution grid features to be leveraged. The results demonstrated superior performance w.r.t. geometry estimation when compared to methods like MonoSDF, and better pose estimation when compared to methods like NICE-SLAM.

Strengths

[1] The idea to fuse input geometry and estimated geometry, instead of using input geometry as constraints when optimizing estimated geometry, is in general novel and sound. The reasoning is, input geometry from fused depth maps may present high confidence in certain areas, and estimated geometry via MLPs may excel at other regions. By fusing these two instead of estimating all regions with MLPs, the method yields better geometry, and benefits pose estimation in SLAM applications. The insights here would be useful to the community to inspire better ways to leverage input geometry when training MLP-based TSDF fields. [2] Extensive evaluation. The methods evaluates the method w.r.t. geometry estimation with state-of-the-art methods like MonoSDF, which is the main focus of the paper. Beyond geometry, the method also shows improved geometry benefits downstream tasks of pose estimation, and is able show superior performance over SLAM systems with implicit neural representations. Extensive ablation study is also applaudable, which justifies certain design choices, as well as providing visualization of what is learned with the attention weights. [3] The paper is in general well-written and easy to follow.

Weaknesses

[1] Further clarify and justifications are needed in various parts of the paper. [1.1] For example, on why Transformers are not used to model the attentions, the paper gives an unconvincing reason that people may conjecture that the performance improvements are mainly contributed to Transformers. However, it is unfair and premature to say so without providing actual results with a Transformer-base design. With those additional results, we may then get a better idea on how important the architecture design of the attention mechanism is to the performance gain. Additionally, design choices mentioned in Line 165-169 require justifications via ablation study. [1.2] Additionally, on the explainability of the learned attention maps, the paper mentions in Line 246 that, 'some area' are paid more attention to, which is not sufficient. Further analysis onto what part of the input geometry is prioritized, and what part is mostly learned, will give better insights of the attention weights, and better justifies the motivation of the proposed. Without further analysis, the attention machoism behaves more or less as a black box. [1.3] Finally, additional details of the experiment setting are needed. For example, in comparing against MonoSDF, it is not clear whether input depth or estimated depth (with DPT) are used. In experiments where poses are optimized, it is not clear what the initializations look like, what pose representation and regularization are used, and what was the convergence. [2] Paper writing. For example, the introduction section repeatedly mentions the core contribution of 'attentive depth fusion priors', without giving any explanation or hints of what it exactly is. As a result, the contribution #2 is essentially explaining the 'attentive depth fusion priors', however it is not immediately clear that the 'attentive depth fusion priors' mentioned in contribution #1 is basically explained in #2. Additionally, more discussion should go into explaining import baselines like MonoSDF and NICE-SLAM, on how are they different from the proposed method, so as to better clarify the contributions, and to provide further insights on why the proposed method performs better.

Questions

There are questions included in my reviews, mostly in Weakness 1.2 and 1.3. it would be great if the authors can address those questions.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Not applicable. No potential negative societal impacts are mentioned.

Summary

This paper aims to improve the performance of 3D reconstruction from multi-view RGB-D images. The key innovation is the attentive depth fusion prior, which allows the networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Experiments show that the proposed method work for both one-time fused TSDF and incrementally fused TSDF.

Strengths

1. This paper aims to improve 3D reconstruction by better handling cases such as incomplete depth at holes. This is one limitation of existing work and is an important task. 2. The method section is clearly and well explained to help readers understand the details. 3. The authors have included video results that can help better understand the visual result quality of the proposed method. 4. Extensive comparison and ablation studies have been done to show improvements over prior work and justify the decision choices of all components.

Weaknesses

1. L242 claims that the proposed method "plausibly completes the missing structures from TSDF" in Fig. 2. However, the completed surface by the proposed method seems to be of very low-quality. It seems a simple non-learning mesh hole filling algorithm on top of the meshes on the left of Fig. 2 can produce much cleaner results. Learning-based methods [A,B] that predict TSDF for indoor scene completion can also achieve much better results. [A] Dai et al. SG-NN: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. CVPR 2020. [B] Dai et al. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. CVPR 2018. 2. It is great that the paper has conducted extensive quantitative experiments. However, there are currently 12 tables in the main paper which seems a bit overwhelming. Indeed, some tables are not that critical and can be moved to supplementary. Alternatively, the authors can only present the scene average evaluation in the main paper for some table and move the per-scene results to the supplementary. The new empty space can be instead used to address some limitations.

Questions

1. Since the proposed method is predicting multiple values for the same query points, what is the time increase over a naive neural implicit function (predicting a single signed distance for each query point)? 2. Minor editorial suggestions: L173: it would be better to swap "color function c" and "occupancy function f" for better correspondence. L191: includes -> include L303: attentions -> attention

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have not adequately addressed the limitations or potential negative societal impact. The authors could show some failures cases that indicates some common patterns where the proposed method would fail on

We would like to discuss with you

Dear reviewers, Thanks for your time and effort to review our submission. We are grateful for your expertise and value your comments. Please take a look at our response to your questions and our additional tables and figures in the PDF file. If you have any questions, please feel free to let us know. We are more than happy to discuss our work with you. Best, Authors

After rebuttle analysis

Response related to misunderstanding of summary: 1) ì...a particular point on a sampled ray for supervision. î We do not use the interpolated occupancy to supervise the inferred occupancy due to the inaccuracy or incompleteness of TSDF.' ANS: There was no misunderstanding, what was meant from the above line is that TSDF is used for better results. This can be understood from the very next line which states "... used as a scene prior" and not as a supervisory signal. However, I acknowledge the inaccuracy in the pointed out line of my summary. 2) ì... TSDF will explain such points the bestî Fig.3 in our manuscript shows that, within bandwidth, TSDF may describe less (such as areas in blue) due to its inaccuracy. ANS: I would not term the above statement of my review as a misunderstanding. This conclusion was drawn based on the very intent of giving more weightage to in-bandwidth points, as has been stated in L129-130 "Since the TSDF predicts signed distances for queries within the bandwidth with higher confidence than the ones outside the bandwidth, we only use the depth fusion prior within the bandwidth." While the TSDF might describe some parts of such areas inaccurately, its ability to explain the within bandwidth points better is exploited in the paper. Moreover, in Fig 3. , blue and red colors are used to show attention value for "beta" and it is very clear that the attention network is quite rightly giving high values to "beta" for in-bandwidth points. The statement does not tally with claims in the manuscript very well. For example it goes against the very observations the manuscript has provided in L310-313 "We try to use attentive depth fusion priors every where in the scene with no bandwidth. The degenerated results indicate that the truncated area outside the bandwidth does not provide useful structures to improve the performance." From this it is logical to infer that areas within the bandwidth provide useful information to improve performance, which is also the crux of the paper. If not, why even bother to have this distinction between inside bandwidth and outside bandwidth regions and use the in bandwidth points for fusion . The second paragraph of the "Contributions" section of the rebuttal also states the same "...showed that it is great to use attention within the bandwidth, ...". 3) We would like to point out another reviewer sZZAís misunderstanding. We did not directly regress two attention weights but used softmax to produce attention weights (L166-167). Ans: Again there is no misunderstanding here too. I have not suggested anywhere in the review that the paper "regress two attention weights". I think that the authors have confused a rather important question "how would it be any different from simply using operator to combine the two occupancies instead of learning two weights". By this question, I do not mean that two weights are being learnt. It is very clear in the paper that a softmax is used to produce attenion weights. What is important here is the question asked. Response with regard to answers provided for my concerns: 1) my responses to explanation given in "The attention network" segment of the rebuttal: Color coding is not clear in Fig. 1(a). I see only red and blue. However, attention is a real number between 0 and 1. So, I would assume the color coding of Fig. 3 in the paper where red is higher and blue is lower. Looking at Fig 1(a-d), it seems that yes, the attention network does choose the inferred occupancy over the TSDF prior when necessary. The analysis depicted here answers my Question 1. 2) Answers provided for my concern raised in Weakness 3 of my review is not satisfactory. What difference does it make if the holes in Fig 2. are invisible areas due to absence of RGBD scans or absence of depth values in the depth image, what is important here is the inability of TSDF to help in such a circumstance. This is clearly visible from quality of mesh obtained from inferred occupancies in the those areas shown in Fig 2. So my concern raised in Weakness 3 is not answered from the provided explanation. 3) Concerns raised in Weakness 4 and 5 of my review is addressed and i am satisfied. 4) Limitations 1 and 2 are also well addressed. 5) Concerns related to "Error accumulation in TSDF" during online mode: I am satisfied with the explanation and with the decision of the authors to also put them in the discussion section of their revised manuscript. Overall: The rebuttal answers most of my concerns except for Weakness 3, which I believe is a significant weakness of this work. Nonetheless, I acknowledge that there is some merit in this work and appreciate the authors detailed response which includes running more experiments. I would also request the authors to include the qualitative and quantitative analysis provided in the rebuttal for concerns raised in Question 1,3 and Limitation 2 in the revised manuscript.

Concerns addressed

Thank the authors for their reply. In my opinion, the rebuttal has addressed most of the concerns raised by Reviewer X87y.