Normal-GS: 3D Gaussian Splatting with Normal-Involved Rendering

Rendering and reconstruction are long-standing topics in computer vision and graphics. Achieving both high rendering quality and accurate geometry is a challenge. Recent advancements in 3D Gaussian Splatting (3DGS) have enabled high-fidelity novel view synthesis at real-time speeds. However, the noisy and discrete nature of 3D Gaussian primitives hinders accurate surface estimation. Previous attempts to regularize 3D Gaussian normals often degrade rendering quality due to the fundamental disconnect between normal vectors and the rendering pipeline in 3DGS-based methods. Therefore, we introduce Normal-GS, a novel approach that integrates normal vectors into the 3DGS rendering pipeline. The core idea is to model the interaction between normals and incident lighting using the physically-based rendering equation. Our approach re-parameterizes surface colors as the product of normals and a designed Integrated Directional Illumination Vector (IDIV). To optimize memory usage and simplify optimization, we employ an anchor-based 3DGS to implicitly encode locally-shared IDIVs. Additionally, Normal-GS leverages optimized normals and Integrated Directional Encoding (IDE) to accurately model specular effects, enhancing both rendering quality and surface normal precision. Extensive experiments demonstrate that Normal-GS achieves near state-of-the-art visual quality while obtaining accurate surface normals and preserving real-time rendering performance.

Paper

References (59)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer Gwpd6/10 · confidence 4/52024-07-08

Summary

The authors propose a novel appearance modeling technique for an chor-based 3D Gaussian Splatting representation based on normal information and an incident lighting parametrization with MLPs. In addition to representation several regularization techniques are used to stabilize normal and incident light optimization. The central claim of the paper is to achieve competitive the rendering quality while achieving more detail geometric information. The authors show experiments on the MIP-NeRF360, Tanks&Temples, Synthetic-NeRF and the deep blending dataset and provide a quantitative comparison in rendering quality and normal-accuracy.

Strengths

1) The paper is easy to read and all components are well explained. 2) The conducted experiments use the most relevant datasets and metrics to assess the image quality, LPIPS, SSIM, PSNR. 3) The underlying research problem of reconstructing accurate appearance and geometry is relevant for the field of neural rendering and an very active research area.

Weaknesses

1) The experimental evaluation only contains 3DGS based methods. Even though there are inherent advantages of 3DGS methods, I'd still expect a comparison to NeRF-based methods on the same research problem, e.g. Ref-NeRF and follow-ups. The central question here would be, how good is the proposed method compared to the best Neural Field based method? 2) The general definition of normals in the context of 3D gaussians sounds a bit vague. The authors say that they define the normal of the a gaussian primitive as the shortest axis of the 3D gaussian without actually defining a surface. How do you define the normals when gaussians overlap or are semi-transparent? 3) The authors claim to improve geometric quality and show the comparison in normal accuracy in table 2. From a 3D reconstruction perspective, the normal accuracy indeed can indicate high quality, however metrics on the reconstructed geometry, like chamfer distance, F1-score, might be a better indicator for geometric quality.

Questions

I'd would be great if the authors could provide explanations to weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors addressed valid limitations in the dicussion part.

Reviewer 68er6/10 · confidence 4/52024-07-09

Summary

This paper proposes a novel appearance modeling of 3D Gaussian Splatting (3DGS) for both accurate appearance representation and geometry reconstruction. Existing 3DGS methods suffer from the trade-off of the accuracy of appearance and geometry due to the disconnection between surface geometry and rendering. To address this problem, this paper reformulates the rendering equation for fast physically-based 3D Gaussian (Normal-GS) rendering, directly connecting surface normal and rendering color. For stable optimization, Normal-GS uses anchor-based MLPs that implicitly consider the local smoothness of local illumination. The experimental results show that Normal-GS reconstructs accurate surface normal while retaining the rendering quality of 3DGS.

Strengths

+ Propose a novel physically-based rendering method for 3D Gaussian Splatting that can use shading cues for normal estimation, backpropagating photometric loss to the surface normal of 3D Gaussian. + Exploit anchor-based 3DGS to represent and stably optimize local incident light without time-consuming ray tracing. + Experimentally show that the proposed method can achieve both high-quality rendering and accurate normal estimation even in complex lighting and specular scenes where existing methods degrade.

Weaknesses

- The illumination of diffuse and specular components is independent of each other. This is physically implausible. - The anchor-based regularization would prevent the reconstruction of detailed geometry like bicycle spokes in Fig. 6. Furthermore, since IDIV depends on the surface normal through $\Omega^+$, it also requires the smooth surface normal.

Questions

- How plausible are the optimized diffuse and specular reflection components? What if only either of them is rendered? - Can the IDIV be visualized to validate that it captures local incident light? For example, in outdoor scenes, it is oriented to the direction of the sun and other directions outside and within the shadow region, respectively. - Scaffold-GS uses the enhanced view-dependent features $\hat{f}_v$ instead of directly using a local feature $f_v$. Which features does this method use? Regarding both geometry and appearance modeling, view-independent Gaussian attributes would be desirable.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes.

Reviewer cnbL7/10 · confidence 4/52024-07-09

Summary

Normal-GS successfully combines color estimation with surface normal optimization, achieving remarkable surface normal prediction without sacrificing view synthesis quality compared to previous methods. In calculating diffuse color, the traditional method that extracts the surface normal components from the incident light integration is used, thus diffuse color is expressed as a simple product of the diffuse albedo, normal, and integration of the incident light (referred to as IDIV in the paper). Based on Scaffold-GS, which predicts multiple nearby 3DGS parameters by passing a single anchor neural Gaussian to a global MLP, IDIV is calculated for each anchor and used for diffuse color prediction. Additionally, by integrating the IDE proposed in RefNeRF into Scaffold-GS, the specular component is successfully represented with respect to the normal. Finally, geometry is regularized by comparing the rendered surface normal with the normal obtained from the depth map.

Strengths

The idea of approximating diffuse color equation in terms of surface normal by extracting the normal from the integral equation is both simple and powerful. Additionally, the geometry inaccuracy that can occur due to view-dependent effects is effectively handled by using IDE to manage specular artifacts. The proposed method compares well with other recent methods that demonstrate robust surface normal in view synthesis. Detailed explanations of IDE and surface normal estimation are well-documented in the supplementary materials, making it easy to follow.

Weaknesses

Due to the lack of explanation of Scaffold-GS in the related work section, the description of the network output is confusing. While it is unnecessary to provide additional details about Scaffold-GS, it is important to clearly explain the network output. As I understand it: In Scaffold-GS, an anchored feature f_v is fed into a global MLP to initially predict the parameters of ‘k’ adjacent 3DGS. In Normal-GS, according to line 194, global MLP theta_l is used to additionally predict IDIV. For diffuse color, it is directly calculated using the predicted IDIV and diffuse albedo. For specular color, it is calculated by passing IDE, normal, and feature into the color MLP theta. However, there is no mention of how components like opacity and diffuse albedo are calculated. It is unclear whether these components are embedded and stored like the original 3DGS, or if they are calculated through a separate MLP. Therefore, it would be helpful to specify which MLP is responsible for each component, similar to IDIV, to clarify the process. There is no quantitative ablation results. Additionally, there is no ablation of the depth-normal loss. Personally, I think this paper, due to its contributions through the refactorization of the rendering equation, would be more suitable for a computer vision or graphics conferences.

Questions

How is "3DGS w/ IDIV" in the ablation studies (section 4.2) (b) implemented? Did you create an additional anchor neural Gaussian to the original 3DGS and use an global MLP to predict IDIV, or did you place it as an optimization component like opacity in each 3DGS and use it for rendering? Additionally, in (c), "ours" generally refers to the complete Normal-GS, so the meaning of "w/ IDIV" is unclear. If (c) refers to Normal-GS without the specular component, it would be better to label it as "Ours w/o L_s." In Figure 2, it seems that there is a typo where the representation of the "vector" in the "IDIV vector" is duplicated.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

As mentioned in the paper, learning the depth of distant objects, like the sky, consistently is extremely challenging. Therefore, it seems that learning surface normal based on depth will also be very difficult. Additionally, as described in the weaknesses section, clearer network outputs would be beneficial.

Reviewer cnbL2024-08-11

I somewhat agree with the comment from reviewer 5i8r that the motivation for directly optimizing the surface normals, which are byproducts of geometry, without targeting the geometry itself, is weak. From the perspective of geometry estimation, I believe that comparisons with methods like SuGaR and 2DGS, which were conducted in the rebuttal, should be included in the paper. However, since the calculation of normals is essential for using geometry as a rendering component, I still believe this research holds value. Assuming that the implementation details and comparisons with existing surface reconstruction method are supplemented, I will maintain my score.

Authorsrebuttal2024-08-12

Thanks for your feedback.

Dear Reviewer cnbL, We are glad to include the implementation details, suggested quantitative ablations, and comparisons with existing surface reconstruction methods. We sincerely appreciate your constructive comments and feedback, which have significantly enhanced the quality and completeness of our paper. Thank you again for your time and efforts during the reviewing process. Best Regards, Authors of Submission 8220

Reviewer 5i8r3/10 · confidence 5/52024-07-12

Summary

This paper addresses the challenge of achieving high rendering quality and accurate geometry in computer vision and graphics. While recent advancements in 3D Gaussian Splatting (3DGS) have enabled real-time high-fidelity novel view synthesis, the discrete and noisy nature of 3D Gaussian primitives hinders accurate surface estimation. The authors propose Normal-GS to integrate normal vectors into the 3DGS rendering pipeline. Surface colors are re-parameterized as the product of normals and a specially designed Integrated Directional Illumination Vector (IDIV).

Strengths

- The writing of this paper is good. It is easy to follow and understand the technical details. - The usage of PBR for optimizing normals of 3DGS is interesting, though it is not new. - Although the experimental results show only slightly better quantitative measurements compared to SpecGaussian, the quality of the normals is significantly improved.

Weaknesses

- L102-103, the authors claim that PBR-based 3DGS methods show lower rendering quality than original 3DGS, which is not correct, please refer to the tables in GaussianShader [7]. - The most closely related work to this paper is 2DGS [44], but it is surprisingly not mentioned or included in the experiment section, which is quite perplexing. 2DGS simplifies 3DGS by aligning it with the surface normal direction, allowing the scale parameters to be directly optimized based on the appearance loss. This omission raises confusion since 2DGS has direct relevance and could potentially provide insights or comparisons for the proposed method. - The integration of Physically Based Rendering (PBR) techniques, specifically Bidirectional Reflectance Distribution Function (BRDF), with 3D Gaussian Splatting (3DGS) is not a novel concept. As mentioned by the authors, previous works have already explored this combination and achieved satisfactory results. Moreover, existing methods mostly enable "relighting" while the proposed method is not. In the proposed method, the authors introduce a simplification of the integral over the up half-space of a point, referred to as IDIV, which represents the relationship between illumination and incident light direction. They decode this IDIV using a Multi-Layer Perceptron (MLP) under the assumption of Lambertian reflectance. However, the authors also draw inspiration from Ref-NeRF and incorporate a specular component to account for specular effects. This additional complexity and the combination of different techniques have resulted in confusion and difficulty in understanding the underlying purpose, raising doubts about the technical novelty of the method, as it appears to be a combination of existing approaches (referred to as an "A+B" manner). - In order to assess the quality of the geometry (indirectly optimized by normals, which is the motivation stated by the authors), there are established metrics and datasets available, such as DTU, that can be utilized. It is perplexing that the authors solely evaluate the Mean Angular Error (MAE) of normals, as the quality of normals is not always the primary factor when evaluating geometry. Instead, the overall geometry quality holds greater importance and should be the focus of evaluation.

Questions

I have expressed my concerns regarding the paper above, and I strongly recommend that the authors address them by enhancing the motivation, technical contributions, and exposition. These improvements are necessary to enhance the quality of the paper. As it stands, I am inclined to reject the current form due to the unclear motivation, absence of baseline methods, and limited applicability. I am also open to revising my rating based on the response and other reviewers' feedback.

Rating

3

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

NA

Authorsrebuttal2024-08-10

Looking forward to your valuable feedback

Dear Reviewer 5i8r, Thanks again for your thoughtful review, which helped us improve the quality and clarity of our paper. We sincerely hope that our rebuttal has addressed your questions and concerns. And if it is possible, please let us know if there are any additional clarifications that we can offer. We appreciate your valuable suggestions. Thank you very much for your time, Best Regards. Authors of Submission 8220

Authorsrebuttal2024-08-10

Looking forward to your valuable feedback

Dear Reviewer cnbL, Thanks again for your thoughtful review, which helped us improve the quality and clarity of our paper. We sincerely hope that our rebuttal has addressed your questions and concerns. And if it is possible, please let us know if there are any additional clarifications that we can offer. We appreciate your valuable suggestions. Thank you very much for your time, Best Regards. Authors of Submission 8220

Authorsrebuttal2024-08-10

Looking forward to your valuable feedback

Dear Reviewer 68er, Thanks again for your thoughtful review, which helped us improve the quality and clarity of our paper. We sincerely hope that our rebuttal has addressed your questions and concerns. And if it is possible, please let us know if there are any additional clarifications that we can offer. We appreciate your valuable suggestions. Thank you very much for your time, Best Regards. Authors of Submission 8220

Reviewer 68er2024-08-10

I appreciate the authors addressing all my questions. I raise my rating from Borderline Accept to Weak Accept.

Authorsrebuttal2024-08-12

Thanks for your feedback.

Dear Reviewer 68er, We appreciate your time and valuable feedback during the reviewing process. We believe your suggestions help improve the clarity and plausibility of our paper. We will include the suggested results and clarifications in our final version. Best Regards, Authors of Submission 8220

Authorsrebuttal2024-08-10

Looking forward to your valuable feedback

Dear Reviewer Gwpd, Thanks again for your thoughtful review, which helped us improve the quality and clarity of our paper. We sincerely hope that our rebuttal has addressed your questions and concerns. And if it is possible, please let us know if there are any additional clarifications that we can offer. We appreciate your valuable suggestions. Thank you very much for your time, Best Regards. Authors of Submission 8220

Reviewer 5i8r2024-08-11

I appreciate the authors' efforts during the rebuttal phase. After carefully reviewing their responses and the comments from other reviewers, I believe my initial justification is correct and valid, and I stand by my evaluation leaning towards rejection. IMO this work still requires further improvement to meet the standards of NeurIPS, including the motivation, technical novelty, experiments, etc. This paper may be better suited for graphics venues instead of machine learning conferences. Some specific comments: * I disagree with the assertion that the proposed method surpasses the inverse rendering technique for its 'environment map representation.' The proposed method also utilizes PBR-based rendering, albeit in a simplified form without the functionality of relighting or material editing. Furthermore, inverse rendering methods demonstrate better performance in novel view synthesis compared to 3DGS. * I maintain my evaluation that this work is incremental, as there are already numerous existing studies on 3DGS employing PBR rendering with simplified BRDF, such as split-sum. Recent works, such as [1][2], have incorporated more accurate PBR representations within 3DGS, and I think this direction is more promising as it significantly improves the accuracy of PBR rendering in 3DGS. [1] Unified Gaussian Primitives for Scene Representation and Rendering [2] 3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes

Authorsrebuttal2024-08-12

Thanks for your feedback.

Dear Reviewer 5i8r, Thank you for your feedback and the opportunity to address your concerns. We fully acknowledge the significant contributions of previous PBR methods [7, 16, 17, 45] targeting inverse rendering, and we appreciate your references to recent methods combining ray tracing with 3DGS. Our approach, however, addresses **the core balance issue between rendering and geometry quality** through a simple yet effective design, without complex regularizers, assumptions, or ray tracing, as detailed in our paper (L32-36, L52-53, Figure 1) and response to Q5. This focus differentiates our approach from inverse rendering techniques that often require additional regularizers or assumptions. In the Experiment section of our paper, we directly compared our method with a prior inverse rendering method [7] and showed our superior rendering quality and normal accuracy. We believe our method establishes a new baseline that could benefit the community. We respectfully emphasize that this explanation is crucial to understanding the motivation and advantages of our method. Regarding the “environment map representation” and “split sum” [A], both of which assume the existence of a global environment map, we would like to clarify that our method does not use this representation. Instead, as comprehensively outlined in our main paper (L165-174, Figure 4, experiments) and our response to Q1, this assumption has limited applicability in real-world scenarios. To clarify further, Figure 4 clearly illustrates the disadvantage of relying on global environment maps, where [7] fails to capture specular effects but our method succeeds. Moreover, in the “Dr. Johnson” scene in “Deep Blending”, which involves multiple rooms without a global environment map, the previous method [7] failed, as shown in Table 1 of our paper. We appreciate the suggestion that our work might be “better suited for graphics venues.” However, given the success of related work in the field of neural rendering and geometry modeling, such as IDR [B], Neural-PIL [C], NeuS [D], VolSDF [E], SAMURAI [F] and NDRMC [G] at NeurIPS, we believe our contributions align well with the conference’s scope, particularly in “Machine Vision.” Thank you again for your thorough review. Best regards, Authors of Submission 8220 [A] Brian Karis. Real shading in Unreal Engine 4. SIGGRAPH 2013 Course: Physically Based Shading in Theory and Practice, 2013. [B] Yariv, Lior, et al. "Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance." NeurIPS (2020). [C] Boss, Mark, et al. "Neural-pil: Neural pre-integrated lighting for reflectance decomposition." NeurIPS (2021). [D] Wang, Peng, et al. "Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction." NeurIPS (2021). [E] Yariv, Lior, et al. "Volume rendering of neural implicit surfaces." NeurIPS (2021). [F] Boss, Mark, et al. "Samurai: Shape and material from unconstrained real-world arbitrary image collections." NeurIPS (2022). [G] Hasselgren, Jon, et al. "Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising." NeurIPS (2022).

Reviewer Gwpd2024-08-12

Dear authors, Thank you , I appreciate your efforts in answering my concerns. The provided results show that the actual reconstructed geometry is accurate and I strongly encourage the authors to add the answer to Q3 to the main paper. The comparison against the NeRF baslines provide evidence that it is also competitive to this line of works. AS the authors resolve my main concerns, I increase my rating to weak accept. Best Reviewer Gwpd

Authorsrebuttal2024-08-12

Thanks for your feedback.

Dear Reviewer Gwpd, We are glad to include the suggested measurements of the overall geometry quality, clarifications regarding the normal definition, and comparisons to NeRF-based methods. Your valuable feedback has significantly improved the quality and completeness of our paper. We thank you again for your time and efforts during the review process. Best Regards, Authors of Submission 8220

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC