Although 3D Gaussian Splatting has been widely studied because of its realistic and efficient novel-view synthesis, it is still challenging to extract a high-quality surface from the point-based representation. Previous works improve the surface by incorporating geometric priors from the off-the-shelf normal estimator. However, there are two main limitations: 1) Supervising normals rendered from 3D Gaussians effectively updates the rotation parameter but is less effective for other geometric parameters; 2) The inconsistency of predicted normal maps across multiple views may lead to severe reconstruction artifacts. In this paper, we propose a Depth-Normal regularizer that directly couples normal with other geometric parameters, leading to full updates of the geometric parameters from normal regularization. We further propose a confidence term to mitigate inconsistencies of normal predictions across multiple views. Moreover, we also introduce a densification and splitting strategy to regularize the size and distribution of 3D Gaussians for more accurate surface modeling. Compared with Gaussian-based baselines, experiments show that our approach obtains better reconstruction quality and maintains competitive appearance quality at faster training speed and 100+ FPS rendering.
Paper
References (65)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper presents a method for improving geometry reconstruction for 3D Gaussian Splatting. It adopts a relatively flattened 3D Gaussian and incorporates normal regularization from monocular priors. Specifically, the paper proposes supervising the geometry by minimizing the differences between the normals derived from the depth map and the monocular normals. The method further refines the depth map by calculating the ray-Gaussian intersection and introduces a densification technique that splits large Gaussians on the surface into smaller ones, effectively reducing depth errors. To address multi-view inconsistencies arising from monocular priors, the paper proposes a weighting scheme to boost performance. Extensive results show that the method achieves competitive results on common benchmarks, including TNT, MipNeRF360, DTU, and Replica.
Strengths
1. The paper presents a confidence map to resolve the multi-view inconsistencies arising from monocular priors estimated from pre-trained models. 2. The paper identifies that large Gaussians cause significant depth errors and proposes a reasonable splitting scheme to address this issue. 3. The paper is well-written, easy to follow, and the evaluation is comprehensive.
Weaknesses
1. The contributions seem limited. The core component of the paper is the combination of depth normals and rendered normals for supervision with monocular priors. However, the depth normals and rendered normals appear similar to those in 2DGS, despite this paper using monocular priors for supervision between them. The intersected depth also shares a similar motivation to 2DGS and Gaussian Surfels [1]. However, the paper lacks acknowledge and reference to these works. The paper should mention these or provide sufficient discussion. 2. While the introduced monocular normals improve reconstruction quality, they also cause oversmoothing, as shown in the DTU results. Additionally, it remains unclear how the monocular priors work for more general cases. The paper should discuss these as limitation. 3. The idea of using monocular normal prior has been adopted in [1], the paper should have a citation or comparison. [1] High-quality Surface Reconstruction using Gaussian Surfels. In SIGGRAPH'24
Questions
1. The paper states that using the rendered normal for supervision can only update the rotation of the Gaussians. Why? Given that the rendering equation involves all the Gaussians and they are all differentiable, if we use a multi-view normal map and train the Gaussians with the rendered normal map, is it really true that these Gaussians will not be optimized into a shape that fits the given normal maps? 2. Why setting the last scale to zero poses difficulties in optimization (L109)? Many papers have verified that setting the Gaussians to be 2-dimensional does not hurt performance [1][2]. I believe more justification is needed. 3. Why 2DGS is faster than the proposed method in the training but has a lower FPS. The paper states the reason is that 2DGS applies a time-consuming ray-splat technique. However, according to the 2DGS paper, the ray-splat intersection does not seem as expensive, at least not more expensive than in this paper, since they also use intersected depth. Some analysis would be beneficial. [1] High-quality Surface Reconstruction using Gaussian Surfels. In SIGGRAPH'24 [2] 3D-HGS: 3D Half-Gaussian Splatting. arXiv:2406.02720 **Minors**: L175: "splitting" should be "splatting." L198: I am also concerned about the use of "view consistent". In my opinion, the monocular normal priors are still multi-view inconsistent, although the proposed weighting scheme can reduce some negative effects.
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
The weakness has been outlined previously, and I have a few questions that need clarification. I may raise my score if these questions are addressed properly.
Response
Thank you for your response. While the idea of incorporating monocular normal priors and a better densification strategy has potential, the current submission has significant issues that make it difficult to accept: 1. The major insight - supervising normals rendered from 3D Gaussians, which only updates the rotation parameter - is not well-justified. The response (plus the illustration figure in the PDF file) only considers a case optimizing normal from a single view, but the paper uses multiple-view normal maps for supervision. Therefore, the current explanation is not well adapted to multiview settings. Although the ablation study shows that using depth-normal is better than render_normal, I believe that this is because using depth-normal will smooth the depth map, and thus enhance TSDF fusion. This core insight needs stronger justification. 2. The paper fails to adequately discuss related prior work, particularly [1], which has been referenced in the current submission. Several formulations and methods (e.g., D-normal, normalized expected depth, and meshing approach) are identical to those in [1], yet there is no proper citation or discussion. The response claims concurrent development, but [1] was published two months before the submission deadline, and the major tables in this paper are also from [1]. This raises concerns about the originality and independence of this work. The claim of important analysis not found in [1] is unconvincing, as the formulation remains the same, and [1] did not require monocular priors for their analysis. 3. The use of monocular priors, though potentially beneficial, is not sufficiently motivated or insightful in the current manuscript. MonoSDF has successfully applied monocular normal priors to challenging scenarios like DTU (3 views), ScanNet, and TnT advance, whereas the current evaluation on DTU (48+ views), TnT, and MipNeRF360 does not sufficiently justify the need for monocular normal priors. Additionally, the argument that normal consistency loss in [1] can lead to conflicting updates is undermined by the fact that the proposed method (F1 score: 0.37) still performs worse than GOF [2] (F1 score: 0.46), which used normal consistency without monocular priors and was published four weeks before the submission deadline. 4. The response mentions omitting auxiliary renderings such as depth, normal, and semantic maps for efficiency. However, it is unclear whether this omission was also applied to [1], making the efficiency comparison questionable. In summary, the manuscript is not well-positioned, and its major insights are flawed and lack proper justification. The adoption of monocular priors, while not novel, fails to demonstrate significant effectiveness compared to prior work. Therefore, I am leaning toward rejection. With substantial revisions and better positioning, the paper could potentially have a greater impact in the future. [1] 2D Gaussian Splatting for Geometrically Accurate Radiance Fields. arXiv:2403.17888 [2] Gaussian Opacity Fields: Efficient and Compact Surface Reconstruction in Unbounded Scenes. arXiv:2404.10772
## Response to Reviewer 5p3w (R\#4) **Q1**: The major insight \- supervising normals rendered from 3D Gaussians, which only updates the rotation parameter \- is not well-justified. The response (plus the illustration figure in the PDF file) only considers a case optimizing normal from a single view, but the paper uses multiple-view normal maps for supervision. Therefore, the current explanation is not well adapted to multiview settings. **A**: The figures drawn in the single view are simplified to illustrate the optimization of positions of Gaussians under normal and d-normal supervisions. As we said in the rebuttal, supervision on rendered normal maps cannot effectively affect the positions of Gaussians, while our d-normal regularizer can. We show the mathematical proof below. **\*\*Propositions:\*\*** ***\*\*\*Case 1:\*\*\**** Supervision on rendered normal cannot effectively affect the positions of Gaussians ***\*\*\*Case 2:\*\*\**** Supervision on our D-Normal regularizer can effectively affect the positions of Gaussians **\*\*Proof:\*\*** Without a loss of generality, we omit the summation over multiple views in our following derivations for brevity. Based on the loss $\\mathcal{L\_{\\text {n}}}$ on rendered normal (*cf.* Eq. 8 of our paper), the gradient of $\\mathcal{L\_{\\text {n}}}$ with respect to position $\\mathbf{p}$: $$ \\begin{align} \\frac{\\partial \\mathcal{L\_{\\text {n}}}}{\\partial \\mathbf{p}\_i} &= \\frac{\\partial L\_n } {\\partial \\hat{\\mathbf{N}}} \\cdot \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\mathbf{p}\_i} \\\\ \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\mathbf{p}\_i} &= \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial \\mathbf{p}\_i} \+ \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\mathbf{n}\_i} \\cdot \\frac{\\partial \\mathbf{n}\_i}{\\partial \\mathbf{p}\_i} \\\\ &= \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \\frac{\\partial G(\\mathbf{x})}{\\partial \\mathbf{p}\_i} \\\\ &=\\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \[ \-G(\\mathbf{x}) \\cdot (\\mathbf{R} \\mathbf{S} \\mathbf{S}^\\top \\mathbf{R}^\\top)^{-1} \\cdot (\\mathbf{x} \- \\mathbf{p}\_i)\] \\\\ &\\approx \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \[ \-G(\\mathbf{x}) \\cdot (\\mathbf{x} \- \\mathbf{p}\_i)\] \\propto (\\mathbf{x} \- \\mathbf{p}\_i) \\end{align} \\tag{1} $$ Based on the D-Normal regularization $\\mathcal{L\_{\\text {dn}}}$ (*cf.* Eq. 11), the gradient of $\\mathcal{L}\_{\\text {dn}}$ with respect to position $\\mathbf{p}$: \\begin{align} \\frac{\\partial \\mathcal{L}\_{\\text {dn}}}{\\partial \\textbf{p}\_i} &= \\frac{\\partial \\mathcal{L}\_{\\text {dn}}}{\\partial \\bar{\\textbf{N}}\_d} \\cdot \\frac{\\partial \\bar{\\textbf{N}}\_d}{\\partial \\hat{D}} \\cdot \\frac{\\partial \\hat{D}}{\\partial \\textbf{p}\_i}, \\\\ \\frac{\\partial \\hat{D}}{\\partial \\textbf{p}\_i} &= \\frac{\\partial \\hat{D}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial \\textbf{p}\_i} \+ \\frac{\\partial \\hat{D}}{\\partial d\_i} \\cdot \\frac{\\partial d\_i}{\\partial \\textbf{p}\_i} \\\\ &= \\frac{\\partial \\hat{D}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\textbf{x})} \\cdot \\frac{\\partial G(\\textbf{x})}{\\partial \\textbf{p}\_i} \+ \\frac{\\partial \\hat{D}}{\\partial d\_i} \\cdot r\_z \\cdot \\frac{\\mathbf{n} } {\\mathbf{n} \\cdot \\mathbf{r}}. \\tag{2} \\end{align} We can deduce the following from Eq (1) and (2): ***\*\*Case 1:\*\**** From Eq. (1), we can see that the gradient-update $\\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\mathbf{p}\_i}$ of position is independent of the normal $\\mathbf{n}$. Consequently, the ***\*\*\*supervision on rendered normal cannot effectively affect\*\*\**** the Gaussian position $\\mathbf{p}$. ***\*\*Case 2:\*\**** From Eq. (2), there is an additional term with $\\frac{\\mathbf{n}}{\\mathbf{n}\\cdot\\mathbf{p}}$, where the denominator $\\mathbf{n} \\cdot \\mathbf{r}$ is a scalar term. This effectively makes the change in the position $ \\frac{\\partial \\hat{D}}{\\partial \\mathbf{p}\_i}$ to move along the direction of the normal $\\mathbf{n}$. Consequently, the ***\*\*\*supervision on D-Normal directly affects\*\*\**** the Gaussian position $\\mathbf{p}$. We can further deduce that the gradient-update on the Gaussian position ***\*\*\*pulls the position along the normal\*\*\**** towards the surface, which achieves better reconstruction. **\*\*(Q.E.D)\*\*** In view of the above proof, we conclude that it is better to do supervision on the D-Normal regularizer.
**Q2**: Although the ablation study shows that using depth-normal is better than render\_normal, I believe that this is because using depth-normal will smooth the depth map, and thus enhance TSDF fusion. This core insight needs stronger justification. **A**: We respectfully disagree that using depth-normal will smooth the depth map, and thus enhance TSDF fusion. From Fig. 1 of the PDF file in our rebuttal, we can see that the proposed D-Normal regularizer effectively pushes the 3D Gaussians towards the surface thus providing much cleaner reconstruction, while only rendered normal supervision cannot. We further provide the mathematical proof in our response to Q1. **Q3**: The paper fails to adequately discuss related prior work, particularly \[1\], which has been referenced in the current submission. Several formulations and methods (e.g., D-normal, normalized expected depth, and meshing approach) are identical to those in \[1\], yet there is no proper citation or discussion. **A**: We have cited 2DGS in our paper and even have done a comprehensive comparison with it on four datasets. We ***\*\*\*did not claim\*\*\**** the expected depth and meshing approach as our contribution. The meshing approach TSDF fusion is a common method used in 3D reconstruction and is not first proposed by 2DGS. We'll make this clearer in our revision. **Q4**: The response claims concurrent development, but \[1\] was published two months before the submission deadline, and the major tables in this paper are also from \[1\]. This raises concerns about the originality and independence of this work. **A:** **\*\*Concurrent Development\*\*** We respectfully request the reviewer to refer to the arXiv version of \[1\], it can be clearly seen that the first version of \[1\] is posted on: [\[v1\]](https://arxiv.org/abs/2403.17888v1) Tue, 26 Mar 2024 17:21:24 UTC (23,221 KB), which is clearly ***within 2 months*** of the NeurIPS submission deadline on 22 May 2024\. Furthermore, it usually takes a few days for the paper to be available after submission to arXiv. It is stated in the “**NeurIPS 2024 FAQ for Authors**” page that: **\*\*What is the policy on comparisons to recent work?\*\*** Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions. Authors are ***\*\*\*not expected\*\*\**** to compare to work that appeared only a month or two before the deadline. \[1\] 2D Gaussian Splatting for Geometrically Accurate Radiance Fields. arXiv:2403.17888 **\*\*Originality and Independence\*\*** We strongly disagree. We ***\*\*\*do not claim\*\*\**** that we are the first to derive the depth normal formulation in our submission. We cited and mentioned in our paper that we are inspired by VLN (\**cf.\** \[55\] in our paper) which shows the depth normal derivation for depth map prediction. We also mentioned in our submission that our contribution lies in our observation that ***\*\*\*supervising the D-Normal with monocular normal prior\*\*\**** leads to a complete update of the geometric parameters in the Gaussians. We emphasized in our rebuttal (*\*cf.\** To all reviewers) that \[1\] \*\*\****does not provide***\*\*\* the insight and analysis that updating the D-Normal would effectively update all geometric parameters in the Guassians. It is clear that \[1\] is ***\*\*\*not aware\*\*\**** of this important finding since they propose a ***\*\*\*weaker normal consistency loss\*\*\**** (\**cf.\** Eq. 14 of \[1\]) that updates the splat’s normal, which can lead to performance drop. We show the ***\*\*\*mathematical proof\*\*\**** in our response to Q1, and we have ***\*\*\*shown experimentally\*\*\**** in Column A vs B of the table in the response to R\#3 that updating the splat’s normal indeed causes a drop in performance. Lastly, we reiterate our other important contributions (*\*cf.\** To all reviewers) on our ***\*\*\*geometric-aware confidence term\*\*\**** and ***\*\*\*densification and splitting\*\*\**** procedures that lead to further improvements in our performance. **\*\*Major tables in this paper are also from \[1\]\*\*** Although we are not obligated to compare with \[1\], we did not ignore \[1\] completely. We still do a comparison and cite \[1\] and show that our proposed method outperforms them in our submission.
**Q5**: The claim of important analysis not found in \[1\] is unconvincing, as the formulation remains the same, and \[1\] did not require normal priors for their analysis. **A**: We respectfully disagree with that ‘The claim of important analysis not found in \[1\] is unconvincing’. 2DGS just claimed ‘By aligning the splat normal with the estimated surface normal, we ensure that 2D splats locally approximate the actual object surface’, and proposed a weaker normal consistency loss. In contrast, we give the insights show an insightful and important analysis on how our D-Normal formulation can lead to a complete and effective update of all the geometric parameters (rotation, scale, and position) of the Gaussian (*\*cf.\** L184-185 of our paper) in comparison with the naive formulation of normal which updates effectively only the rotation and scale (*\*cf.\** Eq. 5 of our paper). More details can be found in the part of ‘To all Reviewers’ at the beginning of our rebuttal. **Q6**: The use of monocular priors, though potentially beneficial, is not sufficiently insightful in the current manuscript. MonoSDF has successfully applied normal priors to challenging scenarios, whereas the current evaluation on DTU, TnT, and MipNeRF360 does not sufficiently justify the need for normal priors. **A**: MonoSDF is a NeRF-based method, and supervision on normals can effectively affect the positions in 3D space (*cf.* L37 to 39 of our paper). Different from NeRF-based methods, the normals in Gaussian-based methods are not determined by the position of Gaussians. As a result, supervising on normal cannot effectively update the position of Gaussian for surface reconstruction. We thus propose to supervise d-normal with monocular normal priors to effectively update the full parameters of the Gaussians. The effectiveness of our D-Normal regularizer can be seen in Tab. 4 A, the performance drops 0.10 F1-score from 0.4 to 0.3 without the D-Normal regularizer on the TNT. We further show the proof of how the supervision on normals is essential for surface reconstruction in our response to Q1. **Q7**: The argument that normal consistency loss in \[1\] can lead to conflicting updates is undermined by the fact that the proposed method (F1 score: 0.37) still performs worse than GOF \[2\] (F1 score: 0.46), which used normal consistency without monocular priors and was published four weeks before the submission deadline. **A**: The argument of: “GOF uses normal consistency loss shows better results than ours with D-Normal regularizer implies normal consistency loss is better than ours” is illogical. The fair comparison is to use the same baseline with normal consistency loss vs. our d-normal regularizer. As shown in Column B in the table of R\#3Q2, replacing our d-normal with the normal consistency in our framework leads to a performance drop from 0.4 to 0.37. This supports our claim that our D-Normal is better than the normal consistency loss. The main contribution of GOF is a new mesh extracting method, and it can be seen as ***\*\*\*complementary to our method\*\*\****. Since our method is to optimize Gaussians close to the surface while GOF extracts the mesh after the optimization, both methods can be used together for better construction. Although GOF is ***\*\*\*marginally better\*\*\**** than us (0.46 v.s. 0.4), it is ***\*\*\*significantly less efficient\*\*\**** at the cost of longer mesh extraction. We use TSDF fusion which costs \~2 minutes while GOF uses multiple Marching Tetrahedra which cost \~50 minutes. The marginal improvement in the performance of GOF comes with a ***\*\*\*large tradeoff in computational time\*\*\****. Furthermore, GOF is put on arXiv on: **\[v1\]** Tue, 16 Apr 2024 17:57:19 UTC, which is less than 2 months from the NeurIPS submission deadline on 22 May 2024\. We thus have no obligation to make comparisons with them in our submission according to the guidelines of NeurIPS (\**cf.\** our response to Q2). \[2\] Gaussian Opacity Fields: Efficient and Compact Surface Reconstruction in Unbounded Scenes. arXiv:2404.10772
**Q8**: The response mentions omitting renderings such as depth, normal, and semantic maps. It is unclear whether this omission was also applied to \[1\], making the efficiency comparison questionable. **A**: As we have shown in our rebuttal, 2DGS is slow mainly because of their rendering method: compute the intersection point first and then use it to get the Gaussian value for rendering. This is different from our rendering method which remains the same as the original 3D Gaussian Splatting. Omitting auxiliary renderings will only increase efficiency slightly. The use of semantic maps is proposed by us to trim mesh for rendering, and it is not present in 2DGS. As shown in the table below, we provide a comprehensive comparison of the rendering speed on Mip-NeRF 360\. From the table, we can see that with depth (without intersection depth) and normal rendering, our method is still faster than 2DGS. This verifies that the rendering of our method which follows the original 3D Gaussian Splatting is faster than 2DGS. | | With Depth \+ Normal Rendering | | Only Color Rendering | | | :---: | :---: | :---: | :---: | :---: | | Method | 2DGS | Ours | Ours | 3DGS | | FPS | 27 | 114 | 128 | 134 |
Issue with the proof.
> From Eq. (1), we can see that the gradient-update of position is independent of the normal. Consequently, the ***supervision on rendered normal cannot effectively affect*** the Gaussian position. This claim is also not correct. The mistake with the provided proof is that the derivative **$dN/d\alpha_i$ depends on the normal vector $n$**. Let's assume the normal is rendered as: $N = \sum_{i}w_i n_i$, where $w_i = \alpha_i \prod_{j<i}(1-\alpha_j)$ and $\alpha_i = o_i G_i(x)$ We then have: $$dN / dw_i = n_i + \sum_{j=i+1}^{N}n_jdw_j/dw_i$$ $$dw_k / dw_i = -(\alpha_{i+1} + \alpha_{i+2}(1-\alpha_{i+1})+....+a_j\prod_{j=i+1}^{k-1}(1-\alpha_j)) = -\sum_{j=i+1}^{k}T_{kj} \alpha_j$$ $$dN / dw_i = n_i - \sum_{j=i+1}^{N-1} n_j * (\sum_{k=j+1}^{N-1}T_{N_k}\alpha_k)$$ $$dN / d\alpha_i = T_i(n_i-\sum_{j=i+1}^{N-1}n_j * (\sum_{k=j+1}^{N-1}T_{Nk}\alpha_k))$$ From this, we can observe that the gradient update is indeed dependent on the normal vector $n$. I hope this correction will help the authors gain a deeper understanding of their method and refine their claims accordingly.
Response to the authors' comments
Thank you for your response and detailed explanation. However, I believe there may be some misunderstandings regarding my initial concerns. First, while I acknowledge that using D-normal improves quantitative results, my primary concern is with the theoretical basis presented in the paper. The central claim that supervising normals rendered from 3D Gaussians only updates the rotation parameter is not entirely accurate. As I previously mentioned, do the authors genuinely believe that using a multi-view normal map to train the Gaussians would not lead to the Gaussians being optimized into a shape that better fits the provided normal maps? The current theoretical explanation does not adequately address this concern. Second, I have no doubt about the effectiveness of using D-normal with monocular priors. However, I emphasize that the submission **lacks sufficient motivation for why monocular priors should be used**. If comparable results can be achieved with simple regularization, what is the justification for incorporating monocular priors? From my perspective, monocular priors seem less effective on common datasets under dense view settings, such as MipNeRF 360, DTU (dense), and TnT. I suggest the paper focus on more challenging datasets, like DTU (sparse), ScanNet, and TnT Advance, where traditional SDF methods without priors typically struggle due to sparse views and large textureless regions. In these scenarios, the proposed method's effectiveness could be more pronounced. Additionally, comparisons should primarily be made with methods that also utilize depth or normal priors. While I agree that the authors are not obligated to compare with concurrent works that appeared less than two months ago, as per NeurIPS guidelines, it would be beneficial to include a discussion on this work. Since the intersected depth, two normals, and the meshing approach are similar to those in [1], a detailed discussion of the differences is encouraged. The authors should emphasize the unique aspects of their method to ensure it is evaluated on its own merits. I also slightly disagree with the authors' claim that they are not required to compare with [1], considering that the baselines of Gaussian-based methods (e.g., TnT and DTU) are produced by [1] and these mesh generations are closely related to [1]. Lastly, while I acknowledge the "geometric-aware confidence term" as effective and interesting, but the incorporation of densification and splitting seems incremental, as this is a known strategy. Additionally, the integration of normal priors and semantic masks to improve performance seems not new to me. Therefore, the paper needs to demonstrate strong results to warrant acceptance, such as by applying the method to more challenging scenes and providing clear improvements over baselines that do not use monocular normals, particularly in highlighted or textureless scenes. However, in the current submission, I find it difficult to perceive significant improvements, such as the over-smoothed results on DTU and comparable results on MipNeRF 360 and TnT when compared to [1] (the original paper, not the reproduction). Based on the above evaluations, I believe that the disadvantages of the current submission outweigh its advantages, which is the primary reason for my recommendation to reject the paper.
## Response to Reviewer 5p3w (R\#4) **Q1.** My primary concern is with the theoretical basis. Do the authors genuinely believe that using a multi-view normal map to train the Gaussians would not lead to the Gaussians being optimized into a shape that better fits the provided normal maps? **A:** With our illustration (*cf.* Fig. 1 of the PDF file) and theoretical proof (*cf.* our response to Reviewer 5p3w (R\#4) Q1), there is ***\*\*\*no reason for us*** ***to doubt\*\*\**** using multiview normal map to train the Gaussians would not lead to the Gaussians being optimized into a shape that better fits the provided normal maps. We reiterate that from our proof shown in our response to Reviewer 5p3w (R\#4) Q1, the gradient of $\\mathcal{L}\_\\mathbf{n}$ w.r.t. the position is not affected by the supervision on rendered normal. This conclusion ***\*\*\*holds true on multiview\*\*\**** since it would be just a summation of all views on the loss function does not change the fact that the gradient w.r.t. position is not affected by the supervision on rendered normal. **Q2**: If comparable results can be achieved with simple regularization, what is the justification for incorporating monocular priors? Monocular priors seem less effective on common datasets under dense view settings, such as MipNeRF 360, DTU (dense), and TnT. I suggest the paper focus on more challenging datasets, like DTU (sparse), ScanNet, and TnT Advance, where traditional SDF methods without priors typically struggle due to sparse views and large textureless regions. Additionally, comparisons should primarily be made with methods that also utilize depth or normal priors. **A：** **\*\*Dense View Setting\*\*** We respectfully disagree with the reviewer’s comment that “monocular priors seem less effective on common datasets under dense view settings, such as TNT”. As shown in Tab. 1 of our paper on TNT, the recent SOTA methods without normal priors such as SuGaR (CVPR2024) and 2DGS (SIGGRAPH2024) show poor performance (0.19 and 0.3 F1-score respectively) while our method with monocular normal priors achieves significant improvement (0.4 F1-score). **\*\*Textureless Scene\*\*** As shown in Fig. 2 of our paper, we have done experiments on textureless scenes (Replica dataset) and achieved a significantly higher F1-score than 2DGS without monocular normal priors (78.17 vs 64.36). **\*\*Sparse View Setting\*\*** We follow the request of the reviewer to do experiments on the sparse TNT dataset with 80%, 60%, 40%, and 20% images. From the table below, we can see that even when on only 20% of training images, our method still outperforms 2DGS (0.35 vs 0.3) with full images for training, which verifies the effectiveness of the monocular normal prior. | Percent | Full | 80% | 60% | 40% | 20% | | :---: | :---: | :---: | :---: | :---: | :---: | | 2DGS* | 0.3 | \- | \- | \- | \- | | Ours | 0.4 | 0.39 | 0.38 | 0.36 | 0.35 | *We omitted the results from 2DGS on lesser than full views since its full views result is already lower than ours at 20% view.
**Q3:** Since the intersected depth, two normals, and the meshing approach are similar to those in \[1\], a detailed discussion of the differences is encouraged. I also slightly disagree with the authors' claim that they are not required to compare with \[1\], considering that the baselines of Gaussian-based methods (e.g., TnT and DTU) are produced by \[1\] and these mesh generations are closely related to \[1\]. We ***\*\*\*strongly disagree with the rejection of our paper due to a concurrent work\*\*\**** \[1\] put on arXiv within two months of the NeurIPS submission deadline (as per NeurIPS’ policy). Furthermore, ***\*\*\*we have shown\*\*\**** comparisons and highlighted our contributions over \[1\] in both our main paper (*cf.* Tab. 1, Tab. 3, Tab. 6\) and rebuttal (*cf.* our responses “To All Reviewers” and “Reviewer 5p3w (R\#4) Q4” under “Originality and Independence”). We summarize ***\*\*\*our contributions over \[1\]\*\*\**** here again: 1. We provide the ***\*\*\*insight and analysis\*\*\**** that ***\*\*\*supervising the D-Normal\*\*\**** would effectively update all geometric parameters in the Guassians. It is clear that \[1\] is not aware of this important finding since they propose a weaker normal consistency loss (\**cf.\** Eq. 14 of \[1\]) that updates the splat’s normal, which can lead to a performance drop. 2. We propose a ***\*\*\*geometrically meaningful confidence term\*\*\**** (*cf.* L187-198 of our paper) to address the inconsistency across multiple views of the normal prior from a pretrained monocular model. 3. We devise a ***\*\*\*new densification\*\*\**** strategy that splits large Gaussians into smaller ones to represent the surface better. In contrast, both 2DGS stopped at regularizing the normals. 4. Different from 2DGS which has to compute the intersection depth/point first and then render a novel view based on the point because of the different splatting method with original Gaussian Splatting, our method only utilizes the intersection depth for training and surface reconstruction and we can discard it for ***\*\*\*faster rendering\*\*\**** during inference. Refer to Tab. 1, 3 and 6 of our main paper and table (shown below) in n7KL (R\#3) Q2 in the rebuttal for the experimental comparisons with \[1\], where we outperform them in all settings. | | A. Ours | B. Ours \+ normal consistency | C. 2DGS | D. 2DGS \+ d-normal | | :---: | :---: | :---: | :---: | :---: | | F1-score ↑ | 0.4 | 0.37 | 0.3 | 0.34 | Additionally, we re-emphasize our last response that we ***\*\*\*do not claim\*\*\**** the expected depth and meshing approach as our contribution. It is meaningless to compare to 2DGS on meshing approach as it is also not first proposed by them. The meshing approach TSDF fusion is a common method used in 3D reconstruction. **Q4**: Lastly, while I acknowledge the "geometric-aware confidence term" as effective and interesting, but the incorporation of densification and splitting seems incremental, as this is a known strategy. Additionally, the integration of normal priors and semantic masks to improve performance seems not new to me. **A**: We respectfully disagree. Although ‘densification’ is already present in all 3DGS-based methods, the densification strategy we proposed is uniquely targeted at minimizing the depth errors that arise from the remandant errors of the Gaussian normals after supervision (*cf*. L205 to L211 of our paper). To this end, we first randomly sample camera views from a cuboid that encompasses the entire scene for object-centric outdoor scenes and from the training views for indoor scenes. Since we aim to densify only the surface Gaussians, we only keep the first intersected Gaussian and discard the rest for each ray emitted from the camera. Subsequently, we densify only those with a scale above a threshold among the collected Gaussians (*cf*. L212 to L216 of our paper). In addition, the splitting strategy is proposed to avoid clustering. Specifically, we split the old Gaussian into two new Gaussian along the axis with the largest scale instead of using the Gaussian sampling with the position of the Gaussian as mean and the 3D scale of the Gaussian as variance (*cf*. L217 to 224). Both these steps lead to significant improvement as shown in Tab. 4E of our paper. To the best of our knowledge, we are first to propose the above mentioned densification strategy. We ask the reviewer to kindly point us to any specific work(s) that shares the same idea as us. We did not claim the integration of normal priors and semantic masks are our main contributions. Please refer to our response in Q3 for the re-emphasis of our contributions.
**Q5**: Therefore, the paper needs to demonstrate strong results to warrant acceptance, such as by applying the method to more challenging scenes and providing clear improvements over baselines that do not use monocular normals, particularly in highlighted or textureless scenes. However, in the current submission, I find it difficult to perceive significant improvements, such as the over-smoothed results on DTU and comparable results on MipNeRF 360 and TnT when compared to \[1\] (the original paper, not the reproduction). **A**: We emphasize again that we ***\*\*\*strongly disagree with the rejection of our paper due to a concurrent work\*\*\**** \[1\] put on arXiv within two months of the NeurIPS submission deadline (as per NeurIPS’ policy). Moreover, we respectfully point out that the reviewer has made a ***\*\*\*factual error\*\*\**** on the claim that our experimental comparisons to \[1\] are not strong in various scene settings. We re-emphasize that ***\*\*\*we have achieved significant improvements\*\*\**** over \[1\] on various settings compared to \[1\] as mentioned by the reviewer. On TNT, we have made significant improvements compared with 2DGS \[1\] and SuGar (0.4 vs 0.3/0.19). Note that the 0.3 F1-score is the official result from 2DGS (not our reimplemented version). On MipNeRF 360, we can only show visualization results since we cannot obtain the ground truth of its mesh. On Replica dataset which has textureless indoor scenes, we have achieved a significantly higher F1-score than 2DGS without normal priors (78.17 vs 64.36) as shown in Tab. 2 of our paper. The above-mentioned results clearly show the outperformance of our method compared to \[1\] under various challenging settings.
Theoretical proof is not correct
It seems the author may not be fully addressing my core concern, which is that their claims are incorrect. Throughout the review process, I have been emphasizing: > First, while I acknowledge that using D-normal improves quantitative results, my primary concern is with the theoretical basis presented in the paper. The central claim that supervising normals rendered from 3D Gaussians only updates the rotation parameter is not entirely accurate.
## Response to Reviewer 5p3w (R\#4) **Q1.** The theoretical proof is not correct. **A**: We respectfully disagree with the reviewer’s proof. Although $\\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i}$ is a vector related to the normal, $\\frac{\\partial L\_n } {\\partial \\hat{\\mathbf{N}}} \\cdot \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i}$ ***\*\*\*is a scalar\*\*\****, which is a dot product between two vectors. Consequently, the gradient of the position is still proportional to $\\mathbf{x} \- \\mathbf{p}\_i$. It means the moving direction of the position is also unrelated to the normal direction. Here is the complete proof: \\begin{align} \\frac{\\partial \\mathcal{L\_{\\text {n}}}}{\\partial \\mathbf{p}\_i} &= \\frac{\\partial L\_n } {\\partial \\hat{\\mathbf{N}}} \\cdot \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\mathbf{p}\_i}, \\tag{1}\\\\ \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\mathbf{p}\_i} &= \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial \\mathbf{p}\_i} \+ \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\mathbf{n}\_i} \\cdot \\frac{\\partial \\mathbf{n}\_i}{\\partial \\mathbf{p}\_i} \\\\ &= \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \\frac{\\partial G(\\mathbf{x})}{\\partial \\mathbf{p}\_i} \\\\ &=\\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \[ \-G(\\mathbf{x}) \\cdot (\\mathbf{R} \\mathbf{S} \\mathbf{S}^\\top \\mathbf{R}^\\top)^{-1} \\cdot (\\mathbf{x} \- \\mathbf{p}\_i)\] \\\\ &\\approx \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \[ \-G(\\mathbf{x}) \\cdot (\\mathbf{x} \- \\mathbf{p}\_i)\] \\tag{2} \\\\ \\\\ &\\text{Putting (2) into (1), we get:} \\\\ \\\\ \\frac{\\partial \\mathcal{L\_{\\text {n}}}}{\\partial \\mathbf{p}\_i} &\\approx \\frac{\\partial L\_n } {\\partial \\hat{\\mathbf{N}}} \\cdot \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \[ \-G(\\mathbf{x}) \\cdot (\\mathbf{x} \- \\mathbf{p}\_i)\] \\\\ &= \\beta \\cdot \\frac{\\partial \\alpha\_i}{\\partial G(\\mathbf{x})} \\cdot \[ \-G(\\mathbf{x}) \\cdot (\\mathbf{x} \- \\mathbf{p}\_i)\] \\\\ &\\propto (\\mathbf{x} \- \\mathbf{p}\_i), \\text{where } \\beta \= \\frac{\\partial L\_n } {\\partial \\hat{\\mathbf{N}}} \\cdot \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i} \\text{is a scalar}. \\end{align} This completes our proof that the gradient with respect to the position is independent of the normal direction.
Response to the further clarification on the proof.
Thank you for the further explanation. I understand that $\frac{\partial N}{\partial \alpha}$ is a scalar, but it is still related to the normal vector, which determines the direction of movement. This suggests that when comparing the rendered normal map with the given normal map, the gradient will indeed influence the position. From my understanding, your proof aims to clarify that this movement occurs along the tangent directions, which I agree with. However, if we use multiple normal maps, would the positions still not be effectively influenced? Another point that seems counterintuitive is that optimizing a scene with normal maps doesn't seem fundamentally different from optimizing one with color maps. Since the latter clearly updates the positions, I'm curious why this would differ when applying normal map supervision.
## Response to Reviewer 5p3w (R\#4) **Q1.** I understand that $\\frac{\\partial N}{\\partial \\alpha}$ is a scalar, but it is still related to the normal vector, which determines the direction of movement. This suggests that when comparing the rendered normal map with the given normal map, the gradient will indeed influence the position. From my understanding, your proof aims to clarify that this movement occurs along the tangent directions, which I agree with. However, if we use multiple normal maps, would the positions still not be effectively influenced? **A**: We respectfully disagree with the reviewer’s comment that “$\\frac{\\partial N}{\\partial \\alpha}$ is a scalar, but it is still related to the normal vector, which determines the direction of movement.” We mentioned that $\\frac{\\partial L\_n } {\\partial \\hat{\\mathbf{N}}} \\cdot \\frac{\\partial \\hat{\\mathbf{N}}}{\\partial \\alpha\_i}$ is a scalar. A scalar value cannot determine the direction of a movement like a vector, it can only affect the amplitude of the movement. Thank you for agreeing “this movement occurs along the tangent directions”. Consequently, the use of multiple normal maps still cannot effectively guide the position of a Gaussian to move toward the surface since the direction towards the surface is absent. **Q2**: Another point that seems counterintuitive is that optimizing a scene with normal maps doesn't seem fundamentally different from optimizing one with color maps. Since the latter clearly updates the positions, I'm curious why this would differ when applying normal map supervision. **A**: We agree that optimizing a scene with normal maps doesn't seem fundamentally different from optimizing one with color maps and the latter clearly updates the positions. However, the target of optimizing one with color maps is to render high-quality images instead of performing 3D surface reconstruction. The movement of the Gaussian positions by optimizing with color maps is for lowering the error between the rendered color and ground truth color. As shown in Fig. 2 (a) of the PDF file in our rebuttal, when the color (normal) of a Gaussian is closer to the GT color (surface normal), the supervision pushes this Gaussian ($\\textbf{P}\_1$) towards the ray to increase its weight in the rendering equation, and vice-versa ($\\textbf{P}\_2$). In a nutshell, this type of position update is only to increase the weight of the ‘correct’ Gaussian, which leads the Gaussian closer to the ray, while decreasing the weight of the ‘wrong’ Gaussian, which lets the Gaussian away to the ray. It is not effective for surface reconstruction, where we want to push Gaussians toward the surface. This is the same as normal map supervision.
Response
Thank you for response. Since that paper primarily uses data (Table) from the 2DGS paper and is closely related, I didn’t initially treat it as concurrent work. After confirming with the ACs and reviewing the submission guidelines, I now believe that [1] should be disregarded in this context. I appreciate the authors' efforts in providing stronger results, and I have no issues with the comparisons and evaluations presented. Therefore, I have raised my score to 4. However, my remaining concern still needs to be addressed. I am willing to increase my score further if this claim is properly resolved.
Some statements might need refinement.
Thank you for your response. I would like to clarify that the scalar of $\partial N / \partial \alpha$ has signs, which allows it to determine the direction of movement, either from $x\rightarrow p$ or $p\rightarrow x$ (pull or push), rather than just the amplitude. Additionally, while the movement occurs in the tangent space, points can still move toward the surface due to simultaneous movement and rotation. Given that in a vanilla approach with pure photometric loss, most Gaussian positions eventually adhere to the surface, I believe this will also be the case with normal map supervision, as the normals will fit the tangent space of the surface. While I find it challenging to reach a full consensus (probably due to the loose definition of "effective" for "surface reconstruction"), I do believe some corrections are necessary: **The normal map supervision can indeed affect the update of positions, so some statements in the paper might need refinement.** Specifically, > L113 DN-Splatter may show severe reconstruction artifacts due to their normal supervision can only update the rotation parameters. > L174 However, the normal is only directly related to the rotation of the Gaussian during Gaussian splitting, which means directly supervising the rendered normals cannot update the positions as shown in Fig. 2. > Figure.2 (Top Right) Although I am not entirely sure I fully grasp its theoretical analysis, I do not see any major issues as long as the suggested corrections are made. I believe that the results are strong enough and really like the geometric-aware confidence term. Therefore, I am raising my score (7) but also lower my confidence level (3).
Thank you for your understanding and discussion. We will refine our paper based on our discussion and your suggestions in the final version.
Summary
This paper presents a confidence-aided Depth-Normal regularizer that directly couples the normal with other geometric parameters, thus enabling the optimization of all geometric parameters from monocular normal priors. The paper also introduces a densification and splitting strategy to regulate the size and distribution of the 3D Gaussian distributions for more precise surface modeling. Experimental results demonstrate that the method achieves better reconstruction quality with faster training speeds and rendering compared to 3DGS and 2DGS.
Strengths
1. The main contribution of the paper is clear. The proposed Depth-Normal regularizer provides a simple but effective way to utilize the monocular normal priors in 3DGS-based reconstruction. 2. Experimental results strongly support the proposed strategy.
Weaknesses
1. The conversion between depth and normals is not a new concept and has been explored in both the field of depth estimation and normal estimation. I suggest referencing some previous works on depth-to-normal conversion, such as VNL[1]. I also acknowledge the contribution of the authors in applying this technique to 3DGS, which allows for better optimization of geometric attributes. [1] Yin W, Liu Y, Shen C, et al. Enforcing geometric constraints of virtual normal for depth prediction[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 5684-5693. 2. 2DGS also proposes a normal consistency loss, which connects the depth and normal. It would be better to clarify the difference and conduct the ablation study comparing the two conversion strategies. 3. Building upon question 2, I'm curious to know if the gain in this paper is primarily due to the introduction of monocular normal priors. For instance, if we maintain the strategy of normal computation in 2DGS and incorporate the monocular normal prior in this paper. Since the major contribution of this paper lies in establishing the correlation between depth and normals, it would be better to provide an explanation or comparison to illustrate the superiority of the used conversion. 4. As shown in Fig.7 and Fig.10, the details of the reconstruction results could still benefit from further improvement. This is also understandable since the reconstruction is based on the monocular normal prior, whose details are limited.
Questions
1. I am curious why training time is longer than 2DGS but inference time is shorter. 2. The reconstruction results of 2DGS in TNT as shown in Fig.4 seems different from 2DGS's original paper in Fig.10, such as the ground of 'truck scene' is not incomplete in 2DGS's paper, the head of 'Caterpillar' does not contain the noisy mesh.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors kindly point out their limitations in the paper. I am curious about the semi-transparent object. Does the monocular normal estimation fails in semi-transparent object?
[ACTION NEEDED] Check author rebuttal
Dear Reviewer n7KL, As we are nearing the end of the author-reviewer discussion period: please take a look at the author rebuttal as soon as possible and see if any of your concerns were addressed. Let the authors know that you read their rebuttal even if it didn't change your opinion of the paper. If your opinion changed, please update your review accordingly. Thank you for your service as reviewer! --Your SAC
Thanks for authors detailed responses and experiments. I have read the rebuttal, and all of my concerns are addressed. I strongly recommend that authors involve the derivation for supporting "the supervision on rendered normal cannot effectively affect the Gaussian position", as mentioned in answer of Q2 and the discussion with reviewer 5p3w. Based on authors rebuttal, I also agree that the reviewer 5p3w's point: "the normal map supervision can indeed affect the update of positions". So some of the claims in the paper need justification. The d-normal achieves better optimization since it optimizes the normal considering the relation between depth and normal, rather than optimize them independently. Currently, the paper is solid and clearly benefits the community by its study in normal supervision of 3DGS. So I am willing to raise my score to accept.
Summary
This paper proposes to reconstruct surface from 3D Gaussians with a view-consistent depth-normal regularization. By introducing normal prior (DSINE/GeoWizard) to regularize the distribution of 3DGS, this approach is able to render smooth and view-consistent depth, facilitating reconstruction. This paper also tries to mitigate the inconsistencies from single-view normal prediction by introducing a confidence map. However, the effect seems to be limited.
Strengths
1) The performance is good. The proposed design can assist 3DGS for better depth rendering. 2) Using the gradient of depth to supervise normal makes sense, which makes the position instead of rotation of each GS effectively optimized and leads to better depth results for reconstruction. 2) The paper is easy to understand and follow.
Weaknesses
1) Actually, using prior from monocular normal estimation has been introduced in several works, e.g. DN-Splatter; Would it help if the d-normal (calculated from simple GS depth / ray-GS intersection depth) is applied? 2) The calculation of confidence is basically the same as D-Norm regularization, according to equation 12 and 13. Typically, in Bayesian learning, we introduce an uncertainty term. It is predicted separately and added in log form as a regularization term to the final loss function. I'm not sure if this approach has been tried before, as the current confidence constraints seem to have limited effectiveness. 3) Some related work should be compared and cited: - Neuralangelo: High-Fidelity Neural Surface Reconstruction - Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing - 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting
Questions
1) I'm not sure why manual normalization is needed in the equation (9), because according to equation (4), the denominator should already be a normalized coefficient.
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
2
Limitations
Similar to many other reconstruction methods, this approach struggles to handle the reconstruction of specular or semi-transparent objects well.
Summary
The paper introduces a novel view-consistent Depth-Normal (D-Normal) regularizer and an uncertainty-aware normal regularizer followed by a new densification and splitting strategy to address the limitations of existing Gaussian Splatting methods in surface reconstruction tasks, such as the supervision of rendered normal updates affecting only the rotation parameter and the inconsistency of predicted normal maps. This approach enables the full optimization of Gaussian geometric parameters and decreases inaccuracy, thereby improving the quality of 3D surface reconstruction.
Strengths
The proposed method introduces novel insights, such as supervising normals rendered from 3D Gaussians, which only updates the rotation parameter. This is quite interesting. The overall model is supported by extensive experiments across multiple datasets, demonstrating significant improvements in both reconstruction quality and rendering speed compared to previous methods. The simplicity and effectiveness of this method are particularly commendable. Additionally, the paper is well-written and presents its concepts in a clear and accessible manner.
Weaknesses
There are not many significant weaknesses in the method. However, a more in-depth analysis of scenarios where the method may fail, particularly with highly inconsistent normal predictions, would provide a clearer understanding of the method's boundaries.
Questions
Please see the weakness section.
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes.
[ACTION NEEDED] Check author rebuttal
Dear Reviewer kSsq, As we are nearing the end of the author-reviewer discussion period: please take a look at the author rebuttal as soon as possible and see if any of your concerns were addressed. Let the authors know that you read their rebuttal even if it didn't change your opinion of the paper. If your opinion changed, please update your review accordingly. Thank you for your service as reviewer! --Your SAC
Question Regarding the Normalization
- According to the derivation of the volume rendering formula, the denominator in Equation 9 ($\sum_i \alpha_i \prod_{j=1}^{i-1} (1-\alpha_j)$) should be a value very close to 1. Although the reported results demonstrate an improvement i due to this normalization, it remains unclear how it works. In the implementation, does the alpha in the denominator participate in gradient backpropagation?
## Response to Reviewer 6weG (R\#2) **Q1**: According to the derivation of the volume rendering formula, the denominator in Eq. 9 should be a value very close to 1\. Although the reported results demonstrate an improvement i due to this normalization, it remains unclear how it works. In the implementation, does the alpha in the denominator participate in gradient backpropagation? **A**: In art, colors can be represented by multiple color overlays, as shown in Eq. 4\. of our paper. However, depth is a geometry attribute in 3D space. It is more suitable to represent depth with the average of depth of Gaussians instead of multiple depth overlays. In the implementation, the alpha in the denominator participates in gradient backpropagation. The equation of the gradient is shown here: **GH**: The denominator in Eq. 9 is very close to 1\. In art and as shown in Eq. 4\. of our paper, colors can be represented by multiple color overlays. However, depth is a geometry attribute in 3D space. It is more suitable to represent depth with the average of depth of Gaussians instead of multiple depth overlays. In the implementation, the ***\*\*\*alpha in the denominator participates in gradient backpropagation\*\*\****. The equation of the gradient is shown here: \\begin{align} \\hat{D} &= \\frac{\\sum\_{i \\in M} d\_i \\cdot \\alpha\_i \\prod\_{j=1}^{i-1} (1 \- \\alpha\_j)}{\\sum\_{i \\in M} \\alpha\_i \\prod\_{j=1}^{i-1} (1 \- \\alpha\_j)} \= \\frac{A}{B} \\\\ \\frac{\\partial A}{\\partial \\alpha\_i} &= \\prod\_{j=1}^{i-1} (1 \- \\alpha\_j) \\left\[ d\_i \- \\sum\_{k=i+1} d\_k \\cdot \\alpha\_k \\prod\_{j=i+1}^{k-1} (1 \- \\alpha\_j) \\right\] \\\\ \\frac{\\partial B}{\\partial \\alpha\_i} &= \\prod\_{j=1}^{i-1} (1 \- \\alpha\_j) \\left\[ 1 \- \\sum\_{k=i+1} \\alpha\_k \\prod\_{j=i+1}^{k-1} (1 \- \\alpha\_j) \\right\] \\\\ \\frac{\\partial \\hat{D}}{\\partial \\alpha\_i} &= \\frac{\\frac{\\partial A}{\\partial \\alpha\_i} \- \\hat{D} \\frac{\\partial B}{\\partial \\alpha\_i}}{B} \\end{align} It is implemented with custom CUDA kernels. The CUDA code below is our implementation. ``` accum_depth_rec = last_alpha * last_depth + (1.f - last_alpha) * accum_depth_rec; accum_depth_rec2 = last_alpha + (1.f - last_alpha) * accum_depth_rec2; dL_dopa += (((c_d - accum_depth_rec) - depth_final * (1 - accum_depth_rec2)) / weight_final) * dL_dpixel_depth; ```
The derivation of gradient and CUDA implementation are right. However, I don't think "overlays" is a reasonable term to distinguish color and depth. Actually, the equation 4 is just a way to calculate the expectation / mean of a variable, whether it's color or depth. I strongly suspect that this difference is due to the influence of the final weight term $weight_{final} = \prod_{i=1}^M(1-\alpha_i)$, which is the residual between 1 and $\sum_i \alpha_i \prod_{j=1}^{i-1} (1-\alpha_j)$ . For color integration/averaging, we add the $weight_{final}$ multiplied by the background color (0 or 1) to equation 4 in real implementation. However, for depth, there is not a corresponding depth value for calculation (or to replace the background color). Therefore, we must discard the $weight_{final}$ and renormalize the remaining weights.
**Q1.** I strongly suspect that this difference is due to the influence of the final weight term $weight\_{final} \= \\prod\_{i=1}^M(1-\\alpha\_i)$, which is the residual between 1 and $\\sum\_i \\alpha\_i \\prod\_{j=1}^{i-1} (1-\\alpha\_j)$ . For color integration/averaging, we add the $weight\_{final}$ multiplied by the background color (0 or 1\) to equation 4 in real implementation. However, for depth, there is not a corresponding depth value for calculation (or to replace the background color). Therefore, we must discard the $weight\_{final}$ and renormalize the remaining weights. **A**: We agree that the reason is the influence of the final weight term $T \= \\prod\_{i=1}^M(1-\\alpha\_i)$. For color rendering, the final rendered color is $\\hat{\\mathbf{C}} \+ T \* bg\\\_color$. However, we need to normalize the remaining weights for depth since there is no background depth.
Final rating
Thanks to the authors for their detailed responses and extensive experiments during the rebuttal period. I have reviewed the rebuttal, including discussions with other reviewers, and my concerns have been addressed. I believe this is solid work that will further contribute to the community, and I will maintain my original rating. However, I also recommend that some claims in the paper require further justification, such as the statement that only rotation parameters are updated, as discussed with reviewer 5p3w. I encourage the authors to incorporate the theoretical analysis provided during the rebuttal period into the main paper or supplementary materials to enhance the readers' understanding.
Thank you for your review. We will incorporate the theoretical analysis and refine some claims in the final version.
Decision
Accept (poster)