Summary
The author proposes an improvement strategy for reconstructing 3D scenes from in-the-wild data based on the latest 3DGS method, primarily addressing occlusion and appearance changes. The main improvements are as follows:
1.Appearance Encoding with MLP: Introduce a Multi-Layer Perceptron (MLP) to encode the appearance of images. This involves two trainable embeddings as inputs: per-image embedding, per-Gaussian embedding, and base color (SH=0). The output consists of color transformation parameters, γ and β.
2.Uncertainty Modeling: Introduce uncertainty modeling to mask occlusions and dynamic objects. The author uses DINOv2 to calculate the feature similarity between the predicted image and the training image, thereby
guiding 3DGS to avoid reconstructing occlusions.
The motivation of the article is relatively clear, the structure is complete, the content is substantial, and the writing is easy to understand with minimal writing issues. The experimental part is also quite comprehensive, which to some extent demonstrates the effectiveness of the proposed improvement method. However, the submission did not include a video, which reduced the persuasiveness of the paper's results in the task of NVS and view transitions.
Strengths
1. The problem that the authors attempt to solve is indeed a real one, and there is a significant demand in the industry for reconstructing 3D scenes from in-the-wild data.
2. The idea of leveraging DINO features for jointly optimizing occlusion is sound and interesting.
3. The paper has a complete structure, with clear language expression and minimal writing issues. The content is well-organized and easy to understand.
Weaknesses
1. The two improvement ideas proposed in the article (MLP color mapping and uncertainty model) are not uncommon, especially in NeRF research. Although 3DGS is still in its early stages, many works such as SWAG, GS-W, and Scaffold-GS also mention appearance encoding and the introduction of pre-trained models. Therefore, I do not see any significant differences between this work and these existing works. Additionally, in the calculation of feature similarity, a pre-trained model was used for fine-tuning, making this work seem more like a combination of existing methods (A+B stitching) rather than presenting original contributions or theoretical derivations.
2. In Lines 126-128, the author mentions that the latest methods (e.g., mip-splatting, Absgs, Gaussian Opacity Fields) are introduced to improve the original 3DGS, indicating that the research in this article is based on a stronger baseline than the original 3DGS. However, in the experimental comparison, it is only compared with the original 3DGS. As far as I know, the methods introduced also aim to address issues with artifacts and floaters. Therefore, it is difficult to determine how much the performance enhancements are due to the improvement strategies proposed in this paper. A comparison with the stronger baseline should be added in Tables 1, 2, and 3.
3. I don't think the comparisons shown in Figures 4 and 5 are very effective. Although there is an improvement in simulating ambient light compared to the original 3DGS, I cannot agree with the statement in the article that "we can adeptly handle changes in appearance such as day-to-night transitions without sacrificing fine details." Clearly, in terms of stone carvings, wall details, and water surface textures, the original 3DGS appears sharper. Perhaps your improvement sacrifices some details but achieves better fitting for appearance changes and smoother view transitions. Unfortunately, you did not submit a video to support this.
4.Some typos：
-Figure2 can be more clear,you can add some legends to indicate the gradient flow. And what does "affi" stands for?
-The Eq.(2) is wrong(incomplete).
-The superscript in Eq.(7) is incorrect:C~
-L.430,the output of MLP is (β,γ) as mentioned in L.134
Questions
1. Related Work (Sec. 2)
In Line 87, you mention two related works, SWAG and GS-W, but the results analysis section does not compare your method with theirs. I understand that they also use MLP to encode appearance. However, what is the most significant difference between your work and theirs? This distinction does not seem very clear.
2. Method (Sec. 3)
1)Regarding Appearance Modeling, do you still use SH=3 to represent the color of Gaussians? Given that neural networks are already being used, why not directly predict the final color like Scaffold-GS? Instead, you predict color transformation parameters. Could you clarify this choice?
2)In Line 155, you mention that "DSSIM and L1 are used for different purposes in our case." However, it is still unclear why one uses "image rasterized without appearance modeling" for calculation, while the other uses the correct appearance for calculation. You also state that DSSIM is more robust to appearance changes and focuses more on structure and perceptual similarity. Shouldn't it be possible to calculate DSSIM on the corrected image?
3)Is Equation (11) the entire loss function term?
4)In Line 220, you mention that "we project them to all training cameras, removing any points not visible from at least one camera." However, my understanding is that the pruning strategy of the original 3DGS should already prune out invisible points. Is this step critical?
3. Experiments (Sec. 4)
1)In Table 1, why are there only two methods compared on this dataset, while more methods are compared on the Photo Tourism Dataset in Table 2?
2)In Table 2, why is there such a significant difference in FPS between your method and 3DGS? I would expect your method to have a similar rendering speed to 3DGS. Is the difference due to the introduction of previous methods to 3DGS (Mip splitting, Absgs, Gaussian Opacity Fields)? These methods might compress and prune the number of Gaussian points, making it an unfair comparison to the original 3DGS.
3)Ablation Study: Sky Handling. You have three improvements, but I haven't seen any ablation experiments on sky handling. Is its impact on the results considered less significant?
4)Can you explain why adding appearance modeling to some datasets in Table 3 actually leads to a decrease in metrics? Based on all the experimental results, can I understand it this way: the methods proposed in this paper are effective primarily under conditions of strong occlusion and significant appearance changes. For minor occlusion or appearance changes, using the methods in this paper may not necessarily yield improvements and could even decrease the performance metrics?
Limitations
The authors have adequately addressed the limitations and potential negative societal impact of their work.