Discussion with Reviewer E3vH
Thanks for your reply. We appreciate your recognition of our work and performance.
Actually, our HDR-GS is not simply adjusting HDR-NeRF to 3DGS. Here are some comparisons in technical details:
(i) Tone-mapping Operation.
The tone-mapping operation of HDR-NeRF converts the HDR color of the sampled 3D points to LDR color following volume rendering along every single ray. Specifically, the HDR volume rendering is formulated as
$\mathbf{I}^h(\mathbf{r}) = \sum_{i=1}^{N} T_i (1 - \exp(-\rho_i \delta_i)) \mathbf{c}^h_i$,
where $\rho_i$ is the volume density at the $i$-th sampled point. $\mathbf{c}_i^h$ is the HDR color at the $i$-th point. $\mathbf{I}^h(\mathbf{r})$ is the HDR color of the pixel where ray $\mathbf{r}$ lands on. The tone-mapping operation of HDR-NeRF converts $\mathbf{c}^h_i$ to the LDR color $\mathbf{c}^l_i$.
3DGS also adopts a similar point-based rendering in rasterization. So directly adapting HDR-NeRF to 3DGS should perform the tone-mapping operation following the HDR rasterization. Specifically, in Eq.(13), the point-based rendering in HDR rasterization is
$\mathbf{I}^{h}(p) = \sum_{j \in \mathcal{N}} \mathbf{c}_j^h \sigma_j \prod _{k=1}^{j-1}(1-\sigma_k)$.
The naive adaptation should perform tone-mapping to convert $\mathbf{c}_j^h$ to $\mathbf{c}_j^l$ along the ray landing on the pixel $p$. However, as compared in the following table, which is copied from the table in `A-2` of our rebuttal for your convenience, we found this adaptation only yields suboptimal results and speed because it can only extract limited color information on a single ray for HDR-to-LDR transformation and needs to compute many times.
|Tone-mapping|Train Time (min)|Infer Speed (fps)|LDR-NE|LDR-OE|HDR|
|:-|:-:|:-:|:-|:-|:-|
|Ray Tracing (HDR-NeRF)|58|78|39.51|34.68|36.12|
|Global Infer (Ours)|34|126|41.10|36.33|38.31|
To address these issues, we design the Dual Dynamic Range (DDR) Gaussian point cloud model that performs a global tone-mapping operation converting the HDR colors of all Gaussian point clouds to LDR at one time to extract more contextual color information and accelerate the inference speed. As shown in the above table, our global tone-mapping leads to improvements of 1.59/1.65/2.19 dB and 48 fps on LDR-NE/LDR-OE/HDR and speed.
(ii) Data Recalibration.
As analyzed in Lines 174 - 181, HDR-NeRF adopts the normalized device coordinate (NDC) system that rescales the coordinates to the unit cube [-1, 1]$^3$ to help stabilize the training. Plus, the data collected by HDR-NeRF does not provide point clouds for the initialization of Gaussian point clouds.
The naive and straightforward adaptation is to randomly init the positions of Gaussian point clouds within the cube [-1, 1]$^3$ and use the NDC with 2D projections to optimize. However, when we try this naive adaptation, the HDR-GS only achieves poor results of 24.45, 25.31, and 23.08 dB on HDR, LDR-OE, and LDR-NE for two reasons. Firstly, the NDC restricts the representing ability and spatial transformation of Gaussian point clouds. Secondly, training the randomly initialized Gaussian point clouds with few views (only 18) leads to an overfitting issue.
To address these problems, we use the SfM algorithm to recalibrate the camera parameters and compute the initial point clouds. A naive method is to use all images to recalibrate, as mentioned in `Q-2` of our response to reviewer `tLXM`. However, this naive recalibration achieves suboptimal results of only 34.21, 36.19, and 33.75 dB on HDR, LDR-OE, and LDR-NE because the light intensity change degrades the accuracy of feature keypoints detection and matching. Hence, we use the LDR images with the same exposure time to recalibrate in Eq.(14), leading to improvements of 4.10, 4.91, and 2.58 dB on HDR, LDR-OE, and LDR-NE.
(iii) HDR supervision.
HDR-NeRF uses the ground truth CRF correction coefficient $C_0$ for HDR supervision. The naive adaptation should also use $C_0$. Yet, we found that training with $C_0$ correction is unstable and the model easily collapses. Besides, $C_0$ is unavailable in real scenes where HDR-NeRF naively sets $C_0 = 0.5$. Yet, this inaccurate $C_0$ may cause color distortion or introduce black spots, as shown in Figure 6. Thus, we use Eq.(16) as the HDR supervision for quantitative evaluation to stabilize the training process. When HDR images are not available in practice, we do not use inaccurate $C_0$ to avoid degradation.
Besides, we resolved the data splitting issue on the real datasets, as mentioned in `Q-5` of our response to reviewer `tLXM`.
3DGS is a great work. Yet, directly applying 3DGS or naively adapting HDR-NeRF to 3DGS does not work well. Our work, as the first attempt, proposes an effective method to explore the potential of 3DGS for HDR imaging. We are glad to share our code, model, and recalibrated data with the community.
Feel free to ask us if you have other questions. Looking forward to your reply.