Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis

Summary

This paper proposes LE3D, an HDR 3D Gaussian Splatting method with Cone Scatter Initialization, Color MLP, and depth distortion. Specifically, this paper introduces the Cone Scatter Initialization to enrich the estimation of SfM. The Color MLP aims to represent the RAW linear color space. The goal of depth distortion and near-far regularizations is to improve the accuracy of scene structure. Experimental results show that the proposed LE3D achieves promising results.

Strengths

1. The motivation of this paper is good. Nighttime 3DGS and HDR 3DGS are very important. 2. The proposed method includes comprehensive experiments. 3. The experimental settings are easy to follow. 4. The video demo in the supplementary material is good.

Weaknesses

1. The authors mentioned, “the SfM estimations based on nighttime images are often inaccurate.” I agree with this point. However, from my understanding, the initial point clouds can be wrong and can be further optimized by the 3DGS training stage. Therefore, I am wondering if the cone scatter initialization is really important. Could the authors provide ablation studies using the RawNeRF dataset? 2. 3DGS highly relies on the initial point clouds. One of the examples in Fig. 1 (a) shows that the input image can be very dark. I am wondering if the authors can get the initial SfM estimations from such inputs. From my understanding, COLMAP cannot work well on these dark inputs. If so, how do the authors get the initial SfM estimations? 3. The authors mentioned, “the SH does not adequately represent the HDR color information of RAW images due to its limited representation capacity.” Why? Could the authors provide any solid evidence? 4. The depth distortion (Eqn. 8) is quite similar to Eqn. 15 in Mip-NeRF 360. How does the proposed depth distortion differ from the depth distortion of Mip-NeRF 360? Could the authors conduct experiments to compare the two depth distortions? 5. The authors seem to overstate their methods. Compared to RawNeRF, LE3D can reduce training time to 1%. However, the fast training speed is not actually due to the proposed components, but rather from 3DGS itself. If we compare it to the original 3DGS, the training speed of LE3D increased from 1.05 to 1.53, which is an improvement of approximately 45.71%. Since this is the first HDR 3DGS paper, I do not want to be too strict. But I suggest that the authors do not overstate the advantages in terms of speed. 6. The quantitative results of the ablation studies should be shown.

Questions

Please see the Weaknesses.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper LE3D proposes training a 3DGS representation with raw images, instead of preprocessed LDR images. This allows more accurate scene recovery in low-light environments and unlocks applications such as HDR rendering or refocusing. While there has been prior work on training neural scene representations with raw images (notably RawNeRF), this work specifically proposes to use a 3DGS representation which allows real-time rendering. To facilitate training a 3DGS representation with raw images, they make several contributions: 1) Cone Scattering Initialization - a method to "densify" the Gaussian initialization to overcome inaccuracies in SFM 2) Replacing the SH in 3DGS with a colour MLP 3) Introducing space carving losses for better geometry recovery (for downstream tasks)

Strengths

Originality & Significance: I like that the paper brings several of the contributions in the NeRF literature to 3DGS, specifically I like the problem being attempted in the paper, I think HDR rendering and training with raw images is an important problem and it is timely to attempt this problem using a 3DGS representation. Losses such as the distortion loss from MipNeRF360 are widely used in the NeRF literature and I think it's valuable to have an implementation of them in the 3DGS context. Not a significant research contribution, but I also really liked the interactive viewer in the supplement, I think it would be fun to play around with the demo and see the weaknesses of the method. Quality: I think the authors are fairly open about the limitations, for example mentioning NeRF-based methods obtain superior sRGB metrics. Clarity: Paper is well organized.

Weaknesses

Originality: I think although useful, the contribution of the work isn't huge, from Table 1, we can see RawNeRF + GS performs quite well already. I also believe the Related Works section is a bit sparse, there has been a lot of work on low-light image enhancement with NeRF, which I think is relevant (even though it's not about HDR rendering), see a couple below: 1) Lighting up NeRF via Unsupervised Decomposition and Enhancement, ICCV 2023 - method for training NeRFs with sRGB images taken in the dark 2) Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption, AAAI 2024 - method for training NeRFs with sRGB images taken in over/under-exposed scenarios Quality: This is my biggest gripe with the work, some of the contributions are included without significant reasoning. Most importantly: 1) The authors introduce the CSI and say it helps due to the misalignments in the SFM estimation. However, I don't think this claim is sufficiently well-proven. Perhaps a denser initialization helps even in cases with perfect cameras? 2) The authors do not justify including the colour MLP, apart from saying it performs better than just using the SH (due to the linear scaling). I think this point is specifically important since I think the MLP is making the method 1.7 times slower in rendering (looking at Table 1). For 1) I am sympathetic to the authors that this might be hard to justify, but I think 2) should be addressed (see also questions). Clarity: There are quite a few typos/grammatical errors, I list just a few below, but I would implore the authors to use a spell check throughout the text: L38: to the 3D world L170: fed L173: set

Questions

1) Have the authors tried just increasing the rate of the Gaussian splitting instead of CSI? Have they tried anything else apart from CSI for the gaussian density problem? 2) Have the authors tried other strategies apart from a colour MLP? Perhaps the SH are also somehow introducing some bounds/weird gradient updates. Here's a thought: in NeRF if you bound your radiance when it shouldn't be, your density would also become heavily affected by the radiance (as it seems is happening in your case without the colour MLP), perhaps it could be something similar going on? How different is the per channel scaling behaving with/without the colour MLP?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

N/A

Summary

LE3D is a novel method for real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes from RAW images, especially for nighttime scenes. It addresses the limitations of previous volumetric rendering methods by introducing Cone Scatter Initialization to improve SfM estimatied pointclouds, replacing SH with a Color MLP to represent in the RAW linear color space, and introducing depth distortion and near-far regularizations to enhance scene structure accuracy. LE3D significantly reduces training time and increases rendering speed compared to previous methods. The model is tested on the benchmark dataset of the existing RawNeRF paper.

Strengths

1) The paper is well written and has a clear structure that aligns to the contribions which makes it easy to read and understand. 2) The methods part is technical sound and details are described.

Weaknesses

1) Instead of using the Spherical Harmonics the authors propose a view-dependence MLP and explain the change that the MLP can better represent the the linear RGB space, however it is unclear to me why the SH are less expressive than a tiny MLP in the linear RGB space. Can the authors provide more insights on this. 2) Insufficient quantitative investigations of the new view-dependence MLP. I'd expect more evaluation of this part, e.g. the varying size of the MLP and different parameters of the SH model. Moreover it would be good to the effect of switching to MLPs on the render time. 3) While the cone scatter initialization leads better rendering, it would be good to discuss how the additional sampling point influence the total number of gaussians. This number becomes relevant for the fps measurements and might reduce speed.

Questions

Please discuss the points mentioned in the weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Sufficiently discussed.

Summary

The authors aim to leverage 3D Gaussian Splatting with a few additions and changes in order to perform HDR view synthesis. The authors propose that with the addition of cone-scattering to the Structure from Motion initialization, replacement of Spherical Harmonics with a simple MLP for color representation, and extra distance regularizations, an HDR scene may be very quickly optimized to accurately render novel views in real-time. The approach is evaluated on the data used by RawNeRF (CVPR 2022).

Strengths

The paper proposes a reasonable set of additions in order to adapt 3DGS for the purpose of HDR scene reconstruction and view synthesis. Though the submission emerges alongside other comparably-aimed papers, the approach proposed is distinct and its authors provide sufficient evidence of the efficacy of their method. - Originality It is probably not a very distant leap to apply 3DGS to avenues of work done on NeRF, though this work does go further to involve a few novel additions to the method being adapted, in order to address shortcomings that would be present otherwise. Other works with the same aim appear to be emerging at this time, though the approach taken by the authors, affecting the gaussian initialization and performing distance regularizations, appears distinct. - Quality With consideration to the supplementary materials, the submission adequately covers most points of concern. Claims made are supported by the provided experimental results. Included ablation study results show qualitative and quantitative contribution for each improvement proposed by the authors.

Weaknesses

- Clarity For the most part, the writing is clear and straightforward. The context for previous efforts improved upon and problems being solved is well provided. Relevant equations are present and adequately described. Implementation and environment details are well expanded upon in the supplementary materials. However, there are a small number of typos remaining in the submission and supplement. Please make sure to correct these. - Significance The paper is one of several that aim to introduce HDR Gaussian Splatting, though with regards to other work released within this month and the last, no code has been released as of yet, so no direct comparisons between results may be made. However, the stated improvements relative to these other works appear to come out on top.

Questions

It is not explicitly written, but could it be appropriate to briefly address directly the tradeoff in FPS/Training time with accuracy in regards to the comparison between the proposed method and compared 3DGS methods?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The supplementary material sufficiently addresses the limitations and potential negative societal impact of the submitted work.