ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction

Summary

The paper presents a learning-based framework on the well-studied neural surface reconstruction problem. The key contribution of this paper is to take the complex photon-particle interaction into account and present a more generalized pipeline rather than relying on volume rendering. The proposed framework released the power of the transformer to achieve enhanced feature representation of sampled points along the ray. Experiments on several popular benchmarks have shown the effectiveness of the proposed approach.

Strengths

(1) The idea of modeling the complex light transport and releasing the flexibility from regular volume rendering is novel and interesting. (2) Overall, the paper is well presented and easy to follow. (3) The paper has achieved state-of-the-art generalizable neural surface reconstruction performance across different datasets.

Weaknesses

(1) (Major) The idea of using 3D feature volumes with hybrid resolution is not new and has been proposed in NeuralRecon (https://arxiv.org/pdf/2104.00681.pdf). Besides, since there are two major differences (the elimination of FPN and the construction of multi-level projected feature maps) between the proposed hybrid extraction and the original one, it is better to make separate ablation studies to further verify the effect of the two variations. (2) (Major) For the ablation study of the occlusion transformer, compared with directly removing this module, a better ablation way is to attend every point’s feature as the input of the key embedding in the self-attention computation. This way ensures a fair setting (roughly same architecture and complexity) and the only difference is whether the later points contribute to the former ones. (3) (Minor) Visual comparison on view synthesis with other methods: since one of the main motivations of this work is to model the photon-particle interaction, I assume a major outcome is more robust rendering against the variations (blur, specular..) from input views. Thus, it is better to show some visual comparison with other baselines (SparseNeus, VolRecon, NeRF…) to verify this point. (4) (Minor)The main diagram (Figure 2) can be displayed in a clearer and more elegant way. Basically the part of transformer details would belong to either occlusion transformer or render transformer, rather than ‘feature fusion’. Besides, there is ambiguity on what the patches with different colors stand for.

Questions

Overall I think the idea of the paper is novel and technically sound, presenting another perspective of the mainstream methods based on volume rendering. Despite some technical concerns listed in weaknesses, now I lean toward accepting this paper. I expect authors to address my concerns by providing more comprehensive experiments to further show the effectiveness of this work.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Authors have discussed their limitations on rendering speed. I think another limitation is that the SOTA neural surface reconstruction method is still only comparable with a classic MVS-based baseline (MVSNet) at this time. But MVSNet and its extensions must be much faster to get a depth map than the rendering-based ones. So there is a long way to go for this area to further release the power of implicit representation.

Summary

This paper proposes ReTR, a new architecture that leverages transformer to replace the traditional volume rendering process. The insight of the paper is that: the traditional volume rendering equation is oversimplified to model photon-particle interaction. Moreover, the color compositing function highly relies on the projected input view colors, and therefore overlooking intricate physical effects. To solve these two limitations, ReTR replaces the volume rendering equation with a render transformer. The attention map can be extracted from the render transformer to synthesize geometry details. An occlusion transformer is further introduced to obtain finer features. Instead of using FPN features and ResUNet, ReTR also utilizes features from different layers to construct multi-scale features. Experiments are conducted on the DTU dataset, Tanks & Temples, ETH3D, and BlendedMVS. The method is compared with state-of-the-art generalizable NeRF methods and generalizable surface reconstruction methods and achieves the best performance among them. Ablation studies also show the effectiveness of the network architecture. Moreover, ReTR surpasses SparseNeuS and VolRecon even without depth supervision.

Strengths

I like the insights proposed in the paper that the traditional volume rendering equation is oversimplified. The solution that utilizes transformers to replace volume rendering is straightforward yet sound and effective. Experiments are exhaustive and validated the network designation. I also like the discussion of interpreting the render transformer the hitting probability.

Weaknesses

More related work should be discussed in Section 2 for generalizable NeRF methods (NeuRays, CVPR22; Generalizable Patch-Based Neural Rendering, ECCV 2022, ...) and neural surface reconstruction methods (NeuS, ...). Notations in Equations (8) and (9) are not well explained, for example, what are $\mathbf{R}^f$ and $\mathbf{R}^{\text{occ}}$, and $\mathbf{f}_i^{\text{occ}}$ did not appear before Equation (10). There is also a typo at Line 232: **tanks and temples** instead of **tanks and templates**.

Questions

Though the proposed method is effective, mathematically, I did not see how the network architecture can model the geometry better than other neural surface reconstruction methods, such as SparseNeuS and VolRecon -- Especially when the network is trained without depth supervision. I would like to see more insights and explanations from the authors in the feedback.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The main concern of this paper is it requires a long training time, e.g. 3 days on a 3090. It comes with a cost when introducing transformers in the network architecture. Generally, it is not a big problem since the method is generalizable.

Summary

The paper proposes a new framework for generalizable neural surface reconstruction, which utilizes the mechanism of transformers to model the rendering process. The authors first derive a general form of generalizable volume rendering based on existing methods and identify its limitations. They then suggest improving upon this framework by introducing a new rendering approach based on learned attention maps and performing over-ray accumulation in feature space rather than color space. Experiments are conducted on four different datasets and compared to recent baselines, demonstrating superior performance in generalizable reconstruction.

Strengths

- The paper is well-written and exhibits a smooth flow. The authors effectively convey the motivation behind their methods, providing necessary background information and comparing against recent baselines. Additionally, the authors ensure that the reader can easily follow the logical progression of the paper. - Section 3.1 presents a general framework that serves as a well grounded basis for existing methods. The authors' identification of limitations within this framework offers valuable observations, contributing to the overall storyline and motivation of the paper. - The results presented in both the main paper and supplementary material demonstrate the superiority of the proposed method compared to the VolRecon baseline. - The authors provided an ablation study on the various components of the method, effectively differentiating their individual contributions and providing a solid understanding of their impact.

Weaknesses

- The section describing the reconstruction transformer appears to be incomplete and confusing due to several missing details and explanations: - It is unclear how equation 6 (and its improvement in equation 10) fit into the general framework proposed in equation 5. This confusion arises because the final MLP from feature to color does not align with the framework. Additionally, the weight function and color function are not explicitly provided. - The FeatureFusion operation is not defined, leaving ambiguity in understanding its purpose and implementation. - The definition of the "meta-ray token" $f^{tok}$ for a scene is unclear, specifically whether it pertains to per-image (per-ray/pixel) or per-scene information, and how it differs from the image features $f^{img}$. - The meaning of $F_i$ in line 161 is not provided or explained. It is crucial for the authors to address these misunderstandings and revisit the missing definitions in order to clarify the concepts. - While it is acknowledged that the general form presented in section 3.1 oversimplifies the modeling of light transport in 3D scenes, it is hard to perceive how the mechanism of cross-attention over ray samples effectively models complex photon-particle interactions. Real interactions typically occur in spatial domains, whereas the suggested approach focuses on interactions over ray samples. It is suggested that the authors either temper these claims or provide further explanation on how their framework accounts for intricate global physical effects that encompass both global and local physical effects. - The qualitative comparison is somewhat limited as it only includes a comparison to VolRecon. It is essential for the authors to provide visual comparisons with other methods as well, particularly SparseNeuS, to offer a more comprehensive evaluation. - The authors have not presented timing evaluations of their method in both training and evaluation scenarios. Given that the limitation section highlights timing as a significant drawback of generalizable methods, it is important for the authors to address this by providing timing evaluations to enhance the paper's completeness.

Questions

- When evaluating the SparseNeuS baseline, did the authors incorporate depth supervision as well? My understanding is that both ReTR and VolRecon utilize depth supervision during training, while SparseNeuS does not. This raises concerns about the fairness of the comparison between these methods. - In Section 3.2, the paper suggests key rendering properties that the system should possess. However, there is no specific mention of the requirement for the weights to sum up to 1, indicating that all rays are eventually occluded. While this property is not explicitly described, it seems to be employed later in the paper using softmax. It would be helpful if the authors provided further clarification on this matter.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed limitations in the supplementary material. However, it is necessary to present the main limitation of efficiency in the main paper, even if briefly in the conclusion section. Additionally, providing quantitative results that demonstrate the tradeoff between the number of parameters and training/rendering time would greatly benefit the presentation of the method.

Summary

This works focus on generalizable asset reconstruction: given a few posed images, predict the 3D representations using a network. Instead of using volume rendering to compute the transmittance, the authors propose to use transformer on the sampled points to compute the weight of each point. Besides, the author also improve the CNN architecture for feature extraction. Extensive experiments are conducted on multiple datasets, demonstrating better performance.

Strengths

1. Better performance compared to previous compared with previous SOTA methods 2. Code is attached and will be release. 3. method is well explained

Weaknesses

1. Compared to previous methods SparseNeus and Volrecon, this work seems somewhat incremental. Major difference is using a hybrid CNN extractor (Fig 3) and a new transformer architecture. The "occlusion transformer" and "render transformer" seems to be two transformers with fancy names, and I don't see significant difference from the transformer in volrecon. Though Volrecon is a CVPR2023 paper, but apparently the authors use its codebase for develop , as can be seen in the code.zip in the supplementary. 2. I don't agree with the "rethink" title, equation5 doesn't make too much sense to me. It's not explained why the equation can satisfy the occlusion-aware, and especially no guarantee of consistency across multi views. For example, for a sampled point $x$, its weight may be $1$ for the ray $r_1$, but may be $0$ for the ray $C(r_2)$ even when $x$ is the nearest point in ray $r_2$. Therefore, I'm not fully convinced with that Eq 5 is a better modelling of the rendering, and the "rethinking" seems be some sort of exaggeration. Furthermore, given the goal of this work is reconstruction, I don't see why loosing the physics constrain in rendering can benefit the learned geometry. ## Justification of rating. 1. Pros: results are solid (multiple datasets, compared with SOTA baselines), code available. 2. Cons: Somewhat incremental, "rethinking" seems a bit exaggeration. Overall I lean to a borderline accept, as no big technical flaws, but I'm not very confident.

Questions

Please check the weakness section. Besides: 1. This may not be the weakness of this paper, but it's common in the research of this topic. When it's called "generalizable", why it can not generalize to the unseen region of the object? For example, given images of the front views of the statue, why not generalize it to the backview in reconstruction. 2. What's the PSNR compared to other baselines

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Discussion of limitation and broad impacts in the supplementary. No license/asset description according to https://neurips.cc/public/guides/PaperChecklist . But I don't penalize it in the rating.

Summary

This paper introduces a interesting solution for volume renderings in generalizable neural surface reconstruction by leverage Transformers to predict depths and colors from feature volumes. The results on sparse view reconstruction prove its useness.

Strengths

The authors identify the limitation and derive a general form of volume rendering and propose ReTR, a learning-based rendering framework utilizing transformer architecture to model light transport. A hybrid feature extractor is also proposed for achieving better performance.

Weaknesses

Can we replace the volume rendering in the optimization-based methods (e.g. NeuS / VolSDF) with the learned Transformer ? Or is the solution only works for generalizable neural surface reconstruction? What is the performance in scene-level sparse view reconstruction, i.e., Replica/ScanNet. Will HybridExtractor also works for volume rendering based methods (e.g. SparseNeuS / VolRecon) ? Why Transformer? Will CNN/MLP also works for this design?

Questions

See the weakness above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See the weakness above.

Thanks for the rebuttal.

Thanks for the rebuttal, most of my concerns are addressed. I tend to keep my score.

Post rebuttal

I want to thank the authors for making an effort in their rebuttal and addressing the reviewers' concerns. The authors addressed most of my concerns. I still don't fully agree with the presented photon interaction over ray samples, and I feel like this discussion is a bit redundant. I lean on keeping my original score, since the paper requires additional clarifications. I suspect the paper requires a big revision to incorporate all the detailed explanations presented in the rebuttal.

Thanks for the rebuttal

Thanks to the authors for the rebuttal. All of my concerns are addressed. I decide to improve my rating to this paper.

Relation to Deep Voxels

The AC would like to ask how much "rethinking" this is, given Deep Voxels (https://arxiv.org/pdf/1812.01024) used a recurrent network (GRU) along the ray or Deep Shading (https://arxiv.org/abs/1603.06078) used a CNN to shade. Both "learn" the rendering. Later, this fell in disfavour for volumes, and was replaced by differentiable non-learned components, popularized in NeRF. Now this submission proposes transformers/attention, a concept more similar to GRU again than the VRE. It feels like we move in circles. This can be okay, but authors have to acknowledge similar previous attempts. So what is different to what Deep Voxels had to offer already 5 ys ago? If it is just, that a Transformer is a better way to learn ray processing than a GRU?