Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections

Thanks for your prompt and valuable response. We address your remaining concerns below: --- **Concern 1**: "It would be more beneficial to implement a version on Scaffold-GS and then conduct a fair comparison with it." **Answer**: Following NeRF-W [1], this work focuses on the reconstructing 3D scene from unconstrained photo collections, such as tourism photos over the internet, where the transient objects appear constantly in the existing datasets. However, we agree including Scaffold-GS as additional baseline will be beneficial to better showcase novelty. ***Implementation details***: Based on the suggestion, we implement an in-the-wild version of Scaffold-GS: (a) For each image, there will be a learnable appearance embedding, and it will be concatenated with original Gaussian features to serve as the input for the color MLP; (b) 2D UNet is leveraged for transient mask prediction and learned in the same way as Wild-GS; \(c\) The appearance embedding should be optimized for each given reference image in the inference stage and this process takes around 5 seconds. Except (b), the entire process is similar to NeRF-W, and we call this version Scaffold-GS-W. We conduct experiments on the six datasets used in the main paper and rebuttal and provide the results below: Results on Main Paper Datasets: | Dataset | PSNR | SSIM | LPIPS | |--------------------|-------|--------|--------| | Brandenburg Gate | 25.55 | 0.9193 | 0.1106 | | Sacre Coeur | 22.77 | 0.8695 | 0.1352 | | Trevi Fountain | 22.08 | 0.7931 | 0.1684 | Results on Additional Datasets: | Dataset | PSNR | SSIM | LPIPS | |-----------------------|-------|--------|--------| | Palace of Westminster | 23.32 | 0.8611 | 0.1792 | | Pantheon Exterior | 23.48 | 0.8637 | 0.1219 | | Buckingham Palace | 24.90 | 0.8890 | 0.1436 | Model Efficiency on Single GPU: | Metric | Value | |-----------------|---------| | Training Time | 0.15 hrs| | Rendering Speed | 192 FPS | ***Observations***: (a) Scaffold-GS-W significantly outperforms 3DGS-AE (in the rebuttal), especially in SSIM and LPIPS, indicating better reconstruction of local textures and structures. (b) There is still a big margin on evaluation metrics between Scaffold-GS-W and Wild-GS, even though Wild-GS is based on the original 3DGS. \(c\) Scaffold-GS-W's inference process is more time-consuming due to the per-image optimization, while Wild-GS offers faster inference by parsing appearance in a single forward pass. (d) Wild-GS keeps the inference speed of 3DGS, while Scaffold-GS-W shows slightly slower rendering on these datasets compared with 3DGS. Given the superior reconstruction capabilities of Scaffold-GS over 3DGS, we believe integrating our hierarchical appearance modeling with Scaffold-GS could further enhance Wild-GS's performance. This potential improvement will be discussed in the conclusion and future work section. All the results above will be included in the paper. [1] NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections --- **Concern 2**: "I am still confused as to why triplane cannot be replaced by hash encoding. While the authors argue that hash encoding cannot be easily processed and predicted by a 2D Network, I believe that it's the same with tri-plane." **Answer**: Triplane can be processed by 2D networks because each plane (i.e., xy, yz, zx) in triplane is a spatially continuous 2D plane that acts as one "2D projection" of the 3D scene. Using a 2D network (i.e., UNet) to create and process these 2D planes is known to be effective in recent works [1-3]. In contrast, hash encoding [4] is inherently a dictionary/look-up table, which does not have a meaningful 2D spatial structure. Therefore, it cannot be processed by a 2D network that relies on spatial operations. [1] 3D Neural Field Generation using Triplane Diffusion [2] Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction [3] RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion [4] Instant neural graphics primitives with a multiresolution hash encoding --- **Concern 3**: "Local appearance is intended to model the specific image's appearance, and I acknowledge its performance. However, the claim regarding "distinct specular highlights and shadows for physical interactions" appears to be overstated." **Answer**: Thanks for your suggestion. We will tone down this statement in the paper. ---

Thanks for all the comments and discussions from all the reviewers. we would like to continue to address the remaining concerns from Reviewer E9p7: --- **R:** "Could you clarify why there remains a significant gap in evaluation metrics between Scaffold-GS-W and Wild-GS? Is it due to the inefficacy of using MLP to model appearance?" **A:** No, Wild-GS also utilizes MLP for appearance prediction. We think directly applying the existing appearance modeling methods used in NeRF (i.e., learnable appearance embedding in NeRF-W) to 3DGS without consideration of the explicit and discrete nature of this new representation is suboptimal, which is the reason why there is a gap between Scaffold-GS-W and Wild-GS. Our appearance modeling pipeline is more advanced and effective, and this is one of our contributions in this paper. --- **R:** "I believe the field of novel view synthesis could benefit from fresh insights. There's been a lot of work published recently, all tackling similar problems with the same datasets, which feels somewhat incremental. Research shouldn't just focus on incremental performance improvements." **A:** First of all, we believe our research is **not incremental**, and we do bring new insights to the field: 1) We are the first method that makes real-time rendering from in-the-wild image set possible, and the fast training and real-time rendering performance of Wild-GS will significantly propell the real-world applications of novel view synthesis from unconstrained photo collections. 2) Our novel appearance modeling pipeline follows the explicit and discrete nature of 3DGS and will inspire the following works on improving the model performance for this task. 3) The extensive experimental analysis and study will serve as a good starting point and expedite the research in this direction. Secondly, new datasets or new tasks are crucial for advancing a specific research field. However, improving existing tasks using established datasets is equally important for solidifying research progress. For instance, the enhanced performance of the YOLO series [1-4] in real-time object detection has significantly propelled the application of vision models in the real world. Similarly, the evolution of CNN architectures from ResNet [5] to ConvNeXT [6] has laid the groundwork for many vision tasks in recognition, detection, and segmentation. Recently, numerous impressive works [7-10] have adapted 3DGS to existing tasks in novel view synthesis, greatly advancing 3D vision. Therefore, we believe that innovative designs that improve model performance on established tasks are still worth exploring. [1] You Only Look Once: Unified, Real-Time Object Detection [2] YOLO9000: Better, Faster, Stronger [3] YOLOv3: An Incremental Improvement [4] YOLOv4: Optimal Speed and Accuracy of Object Detection [5] Deep Residual Learning for Image Recognition [6] A ConvNet for the 2020s [7] Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024) [8] Text-to-3D using Gaussian Splatting (CVPR 2024) [9] Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (CVPR 2024) [10] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (CVPR 2024) --- **R:** "After reading the paper and especially watching the supplementary video, I found the approach rather uninspiring, particularly for someone who's been in this field for years." **A:** As we discussed in the former questions, directly applying existing methods from NeRF to 3DGS is suboptimal, and our Hierarchical Appearance Modeling approach follows the nature of 3DGS and accomplishes more accurate appearance modeling and transfer capabilities than existing methods. Also, the entire Wild-GS pipeline is highly efficient by providing very fast training and similar inference speed with 3DGS, which other existing and concurrent works cannot reach. Therefore, we believe Wild-GS will **inspire the following works on the design of model architecture and appearance modeling pipeline**. --- **R:** "While this method may be somewhat effective, the results aren't surprising and feel incremental, which I believe falls short of the standard for NIPS." **A:** Wild-GS significantly outperforms existing state-of-the-art models on this task. For instance, compared to CR-NeRF [1], **Wild-GS achieves an approximately 3 PSNR increase while reducing training time by 200 times and increasing rendering speed by 10,000 times**. Even compared with concurrent works, Wild-GS still presents better results. Therefore, from the perspective of experimental results, we still think Wild-GS is **not incremental** and will serve as a good foundation for the following works. [1] Cross-Ray Neural Radiance Fields for Novel-view Synthesis from Unconstrained Image Collections (ICCV 2023) --- Please let us know if you have any other technical concerns so that we can address them on time. ---

Dear Reviewer JabQ, We are truly grateful for the time and effort you have invested in reviewing our paper. We have submitted our responses to your comments, along with a PDF file of the rebuttal. If you have any further questions or need additional clarification, please leave us a comment. We are keen to address any concerns during the discussion period to ensure our manuscript aligns with your expectations. Thank you once again for your insightful feedback and guidance. Warm regards, The Authors

Thanks for the authors' reply. My questions are answered. About triplane feature, I agree with Reviewer E9p7 that triplane is limited for large scenes. For example, recent SMERF [1*] also points out the limitation of triplane representation in MERF [2*] for large scenes and thus tries to solve it. Take a city scene for example, where the region of interest may contain many buildings. Using triplane features is certainly insufficient because of occlusions. As suggested by Reviewer E9p7, I think using pipelines like Scaffold-GS is an interesting direction. For the computation complexity, since 3D point cloud of 3DGS is a sparse data structure, many methods [3*, 4*] that are specialized to deal with such sparse voxels can be used to improve efficiency. [1*] Duckworth et al. SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration. SIGGRAPH 2024. [2*] Reiser et al. MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes. SIGGRAPH 2023. [3*] Choy et al. 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019. [4*] Tang et al. TorchSparse: Efficient Point Cloud Inference Engine. MLSys 2022.

Dear Reviewer ZrLe, We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We have submitted our responses to your comments, along with a PDF file of the rebuttal. If you have any additional questions or require further clarification, please let us know. We are eager to address any concerns during the discussion period. Thank you once again for your valuable feedback and support. Best wishes, Authors

Thanks for your comments. Following Ha-NeRF and CR-NeRF, we use the whole image to extract the appearance embeddings in Table 1 of the main paper. Ha-NeRF states that "NeRF-W optimizes appearance vectors on the left half of each test image while Ha-NeRF does not", considering NeRF-W "can not hallucinate new appearance without optimizing during training." Recent methods (Ha-NeRF and CR-NeRF) directly predict the appearance vector/feature instead of optimizing it for each test image so the whole image can be utilized. Since our method also uses direct appearance encoding, for fair comparison, we follow the same setting. However, for the reader's benefit, we will also attach the evaluation results using only half of the image for testing. Here are the comparison results (average value on three extra datasets used in the rebuttal) of Wild-GS using whole and half of the image: | | PSNR | SSIM | LPIPS | |-------|:---------:|:--------:|:--------:| | Whole | 26.4470 | 0.8788 | 0.1415 | | Half | 26.1326 | 0.8747 | 0.1441 | There is a slight performance decrease when only half of the image is used. We will also include the results in the manuscript appendix. If you have any further questions, please let us know. Thanks once again for your efforts.

Thanks. But there are still some ambiguities. I understand that extracting appearance requires a whole image. I only want to know, in Table 1 of the manuscript, in the Brandenburg dataset, do you use half of the image for calculating PSNR SSIM and LPIPS, or the whole image? For example, for a rendered image $\hat{I}$ and the ground truth image $I$. Do you calculate PSNR using $f_{PSNR}(\hat{I}, I$), or do you use $f_{PSNR}(\hat{I}[:, w//2:], I[:,w//2:]$)? Where w is the width of the image, and $f_{PSNR}$ is for calculating PSNR.

Thanks so much for reminding us of this issue. The table we provided in the former comment is not **calculated** by the whole and half of the image. We actually misunderstood your question before, and the results reported in that table are generated by using the whole and half of the image to parse the appearance embeddings. Thus, we say, "There is a slight performance decrease when only half of the image is used." because less information is used. Here are the new results **calculated** by the whole and half of the image: | | | gate | | | coeur | | | fountain | | |-------|:-----:|:------:|:------:|:-----:|:------:|:------:|:-----:|:--------:|:------:| | | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | | Whole | 29.65 | 0.9333 | 0.0951 | 24.99 | 0.8776 | 0.1270 | 24.45 | 0.8081 | 0.1622 | | Half | 29.92 | 0.9354 | 0.0843 | 25.01 | 0.8797 | 0.1207 | 24.31 | 0.8093 | 0.1593 | Similar to previous works, these two results do not significantly differ. However, with a scientific and rigorous attitude, we need to greatly thank the reviewer for mentioning this issue. We will replace the metrics in Table 1 of the main paper with these new results on **the right half of the image** ($[w/2, h]$).

We sincerely appreciate the insightful, objective, and constructive feedback from Reviewer TYNp, and we are grateful for your support in accepting our paper.