Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting

Summary

This paper introduces an anisotropic spherical Gaussian (ASG) appearance field into 3D Gaussian splatting for modeling the view-dependent appearance of each 3D Gaussian, which increases the ability of 3D Gaussian in representing high-frequency information. The key idea of this paper is combining ASG and SH to model the color of each 3D Gaussian. The experiments show the effectiveness of the method in modeling specular highlights and rendering quality.

Strengths

1. An ASG appearance field is introduced to increase the ability of 3D Gaussian in representing high-frequency information. 2. The coarse-to-fine training scheme can effectively eliminate floaters, which is verified by the corresponding ablation studies. 3. Extensive comparison and ablation studies have been done to show the effectiveness of each component in this method.

Weaknesses

1. The method of this paper includes several MLPs, however, the authors didn't report the training time of their model when compared with other methods. I think the training time should be considered for a more fair comparison. If I missed this, please correct me. 2. The structure of 3D Gaussian in this paper is mainly based on Scaffold-GS, anchor-based Gaussian splatting. Therefore, the performance of the method in this paper may be degraded when the vanilla Scaffold-GS cannot perform well, such as the scene dominated by large texture-less regions.

Questions

1. It would be better if the authors could include the training time of their model when comparing it with other methods. 2. If we use RGB color as the diffuse color directly, instead of using SH to model, whether there will be a better or a worse result, since the high order SH is used in 3D-GS to represent high-frequency information.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. The authors discussed the limitations of their work.

Summary

Spherical harmonics-based 3D Gaussian splatting (3DGS) struggles with specular and anisotropic components. To address this problem, the paper proposes adopting anisotropic spherical Gaussians (ASG). However, directly adopting ASG does not demonstrate superior performance in representing specular and anisotropic parts. Therefore, the paper proposes separating diffuse and specular components from color representations and using a feature decoupling MLP to generate colors from ASG features. Through experiments, the paper demonstrates improved ability to represent highly specular parts.

Strengths

**Novelty** The idea of adopting ASG for color representation in the Gaussian Splatting framework is novel. **Performance** The paper demonstrates significant performance improvements across different datasets.

Weaknesses

**Related Works** The related works section could better highlight the differences and advantages of this paper compared to other studies. The paper does not mention other works on specular scenes and objects. For instance, GaussianShader is another 3D Gaussian splatting-based method for specular scenes and objects, yet it is not mentioned in the related works section, even though it is referenced elsewhere in the paper. The related works section should compare this method with prior works, such as SpecNeRF (CVPR 2024), and highlight their limitations and how the proposed method overcomes them, or emphasize the novelty of this paper. **Mathematical Notations** The mathematical notations could be improved. The inner product is represented both by $\cdot$ (Line 145, Eq. 11) and $\langle \rangle$ (Eq. 10). Additionally, $\cdot$ denotes element-wise multiplication (Eqs. 5 and 6), inner product (Line 145, Eq. 11—inside the parenthesis), scalar-vector multiplication (Eq. 11—outside the parenthesis), and scalar-scalar multiplication (Eqs. 4 and 12). At least element-wise multiplication and inner product should use different notations. And using the same notation for different operations within a single equation (Eq. 11) should also be avoided. **Coarse-to-fine Training** Coarse-to-fine training is not a novel approach within the 3DGS framework. For example, “EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS” already proposed a similar coarse-to-fine training approach.

Questions

Line 144: Is $\xi$ really $\mathcal{R}^2$ or is this a typo of $\mathcal{R}$? Lines 185: Could you specifically state what is decoupled through $\psi$. Line 191: There are no equations for pure ASG nor pure MLP. Could you state the equations or clarify how they work? For example, for pure MLP, is $\kappa$ removed from Eq. 10 and only $\Psi(\gamma(d), \langle n, -d\rangle)$ used? Lines 232-234: This seems contradictory to lines 178-179. Could you clarify which one was actually used during the experiments. Lines 242-244 and Tabs. 3 and 4: Based on the explanation, Ours-light refers to 3DGS + ASG, while Ours-w/ anchor refers to Scaffold-GS + ASG. However, it is unclear what the performance version (Ours) refers to. As stated in lines 220 and 221, is the performance version the same as the light version (3DGS + ASG) with a lower threshold $\tau_g$? If so, clarifying this in the experimental section would help readers understand what the performance version is. Fig 6: Lines 181-183 and 277 state that directly using ASG leads to an inability to represent specular and anisotropic components. If “Ours w/o MLP” is the one, clarifying this in the caption could help with understanding. In addition, the meaning of “MLP” in “Ours w/o MLP” is ambiguous. It is unclear whether MLP in Fig. 6 refers only to $\Psi$ (Eq. 10) or both $\Psi$ and $\phi$ (line 233). **Color separation** As shown in Eq. 9, the proposed method separates diffuse and specular components. What happens without color separation? Additionally, what happens if the diffuse part is also represented through ASG? Based on the ASG paper, replacing SH with ASG can improve performance. Therefore, could you provide a reason why the diffuse part still uses SH?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper stated its limitation.

Summary

This paper proposes using Anisotropic Spherical Gaussians (ASGs) as view encoding to enhance the modeling of specular reflections in 3D Gaussian splatting. In addition to Spherical Harmonics (SH) encoded colors, the method additionally queries reflection direction with multiple ASGs to generate a view encoding. Since ASGs can potentially model higher frequency signals, the proposed method improves reconstruction quality in scenarios with strong view-dependence. The paper also introduces minor contributions, such as coarse-to-fine training and ASG compression, to further enhance quality and efficiency. The results demonstrate that the method surpasses all baselines.

Strengths

The idea presented is neat and simple, which is good. Although it is not too surprising that introducing higher frequency view encoding could enhance appearance modeling, it is noteworthy that no one else has systematically explored this idea. This could inspire the community. Thus, there are contributions, though not significant. I also appreciate the effort to improve rendering and memory efficiency after incorporating the ASG model.

Weaknesses

• Contribution: I would not describe this work as "the first to address specular highlights modeling in GS," as some earlier work has also attempted this. It would be better to tone down this claim. • Sum of L1 Norm of Gradients: One improvement in the paper is achieved by adding L1 norm to the gradient accumulation for GS densification, as illustrated in lines 202-214. I don't really follow the intuition behind this design. The original design for densification involves duplicating Gaussians and moving along the gradient direction to reduce image loss from all views. So when different views suggest moving the Gaussian in different directions, these directions cancel out and we don't do densification. This makes sense because it means that the particular gaussian is centered at an optimal location that won't sacrifice the quality of any view. However, I cannot find any physical meaning for using the sum of L1 norms. Although experiments show better performance, it would be helpful to provide an explanation, possibly with a toy example. Could this performance improvement be because the sum of L1 norms tends to be larger and thus more likely to exceed the threshold? Would reducing the threshold have a similar effect? • Lack ablation on the Number of ASGs: An important hyperparameter is the number of ASGs used. Theoretically, this is crucial for balancing efficiency and quality. • Discussion on Shape-Radiance Ambiguity: An important aspect of reflection modeling in radiance fields is dealing with shape-radiance ambiguity. I assume the coarse-to-fine training helps with this ambiguity, but it is not discussed in the methods or experiments. • Missing Comparison with Inverse Rendering GS: Although inverse rendering has a slightly different task than view-dependent appearance modeling, it is a valid approach to solving this problem. Therefore, it is necessary to compare this method with at least one GS inverse rendering method to demonstrate the benefits of using ASG over IR. • Rendering Time Breakdown: It would be helpful to show a breakdown of rendering time for each sub-step to identify the bottleneck preventing the model from achieving a similar FPS as the original GS. • Additional Visualizations: I would like to see visualizations of the learned ASGs and normal maps. • Performance on Ref-NeRF's In-the-Wild Data: I am curious to see how this method performs on Ref-NeRF's in-the-wild data, which contains more challenging mirror-like reflections. • Missing References: Some important citations are missing. Here are a few: ○ Reflection Modeling in 3DGS: § "3D Gaussian Splatting with Deferred Reflection" ○ Reflection Modeling in Point Cloud: § "Neural Point Catacaustics for Novel-View Synthesis of Reflections" ○ Inverse Rendering in NeRF: § "PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing" ○ Reflection Modeling in NeRF: § "NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections" § "SpecNeRF: Gaussian Directional Encoding for Specular Reflections"

Questions

Please see the weakness.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The modeling of complex reflections (e.g., self-reflections, mirror-like reflections) could be further discussed. Also it would be helpful to demonstrate some failure cases.

Summary

This paper presents an approach for reconstruction and view synthesis of scenes that exhibit strong specular/view dependent appearance. In particular, the authors extend the framework of Gaussian Splatting [Kerbl et al. 2023] and Scaffold-GS [Lu et al. 2023], replacing spherical harmonics for parameterizing view-dependent appearance with anisotropic spherical Gaussians [Xu et al. 2013], as well as leveraging a coarse-to-fine training strategy to generally improve performance. The authors carry out qualitative/quantitative evaluation on a variety of datasets, and report improved quantitative performance over nearly every other baseline.

Strengths

The authors show good quantitative performance on a large number of datasets -- the quantitative evaluation in particular is quite comprehensive -- and some of the qualitative results presented in the video/paper are compelling. The design of the method seems sensible, and various components are ablated to show their importance. The largest contribution of the work (and an important one, if accurate) is assembling a system for reconstruction and view synthesis that performs well for challenging view dependent scenes.

Weaknesses

* I felt that the writing quality could be improved. In section 3, I couldn't tell which of the described components were part of the base method, and which were part of method variants (e.g. ours-light, ours w/ anchor). Method training and architecture details were somewhat sparse (only a few details were provided in section 4.1). I was also confused about how the word anisotropy is being used throughout the paper. Typically, it's used to mean view dependence that is not isotropic (e.g. rotationally invariant) -- but it does necessarily imply *high frequency* view dependence. Spherical harmonics are, in fact, anisotropic spherical functions. * I'm not quite sure how to assess the novelty of this paper, and feel that some claims (e.g. this is "the first work to address the specular highlights modeling in 3D-GS") are not quite fair (other works, such as GaussianShader [Jiang et al. 2024] at CVPR 2024 attempt to improve modeling of view dependent appearance in Gaussian splatting). * For many of the real scenes, it's hard to judge the qualitative improvement of this method over baselines (e.g. Figure 8). The video comparisons are nice, but are not provided for all baselines -- I feel that a webpage would've been more effective for showcasing comparisons/improvements. * As far as I can tell, this work does not make many changes on top of existing methods -- the two main changes being using ASGs to parameterize view-dependent appearance, and implementing a coarse-to-fine training strategy. Perhaps I'm missing something, but I'm not sure why these changes should lead to such a large improvement in view-dependent appearance modeling and quantitative performance. For example, are ASGs really responsible for modeling the reflection in the CD in Figure 9? Or is the ability to model this reflection due to Scaffold-GS's view-dependent decoding of Gaussian parameters? When the removal of floaters is shown in the video for the Bonsai scene, is this due to the coarse-to-fine training strategy, or something else (hard to say, because only the baseline 3DGS is shown in the comparison)? In general I don't feel that the relationship between quantitative/qualitative gains and method design are fully justified, but I acknowledge that I could be in the minority here.

Questions

See above.

Rating

6

Confidence

4

Soundness

3

Presentation

1

Contribution

2

Limitations

Limitations are discussed.