3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

Summary

The paper tackles the problem of enhancing 3GS novel views which are far from existing viewpoints (encountered in sparse-view settings), due to insufficient information in under-sampled areas. This work enhances the low quality views using a video diffusion model to maintain multi view and temporal consistency. The authors reformulate the task of 3D-consistent image restoration as video restoration which is again reformulated as a video interpolation tasks between frames. The high quality frames along with the restored images are then used to fine tune the 3DGS. The authors also propose a modified spatial temporal decoder (STD) to reduce artifacts and introduce confidence awareness at both image level (based on distance of novel view from reference) and pixel level (information gain) to minimize the negative impact of generated images.

Strengths

* The paper is well motivated- enhancing low quality 3DGS rendering results for camera viewpoints which are far away * Reformulates the 3D-consistent image restoration task in 3DGS as temporally consistent video generation to leverage video diffusion models * The results look quite impressive and have been tested on a variety of scenes * can be applied to any existing 3DGS models and more generalized scenes * introduces a dataset for evaluation of artifacts in 3DGS (will this dataset be open sourced?) * Sound experimental setup and adequate comparisons with other methods

Weaknesses

* Related work appears to be thin and some of these paragraphs can be more descriptive * Method relies on adjacent views for continuous interpolation * Choice for STD should be quantitatively ablated. Individual parts in STD like color correction should be ablated as well.

Questions

1. Authors describe in L184-186 "Despite this significant enhancement in the quality of the rendered views, we propose to rely more on the reference views rather than the restored novel views when fine-tuning the 3DGS model, since the 3DGS model is highly sensitive to slight inaccuracies in the restored views." I would appreciate a more thorough discussion on this. 2. "modified baselines denoted with ∗ taken from [36]". However, no baseline seems to be marked with * in table 1. This is possibly an oversight and the authors should correct it. 3. What's the adversarial loss described in L179 for STD? 4. On what data is the Temporal Denosing U-Net fine tuned on? Suggestions: Figure 2 can be more interpretable if the right part shows the output of the enhanced views (input from left part); currently there is no 1:1 parity

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are adequately addressed

Summary

This paper utilizes a video diffusion model to enhance 3DGS rendering results. It proposes a 3DGS-Enhancer pipeline to reformulate 3D-consistent image restoration tasks and leverage it to generate high-quality and 3D-consistent images. They have enough experiments to prove the soundness of their method.

Strengths

They didn't use depth information but video diffusion prior to do the task, which is interesting. They also have the spatial-temporal decoder which solves the temporal consistency problem. For fine-tuning, they add confidence constraints on both pixel level and image level to enhance the result. Extensive comparison with state-of-the-art is done, and the results seem robust.

Weaknesses

They can provide more comparisons on ablating the temporal consistency.

Questions

Video Diffusion prior may have motion information of moving objects in the scene. How do we eliminate that effect in enhancing 3DGS?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation is definitely from the pre-trained video diffusion model whether the data is more like novel view synthesis data (static scene) or a sequence of video frames (moving objects).

Summary

This paper presents a novel pipeline aimed at enhancing the quality of 3D Gaussian splatting (3DGS) representations, especially in scenarios with sparse input views. They propose a novel framework, 3DGS-Enhancer, that leverages video LDMs for generating high-quality and 3D-consistent images. Moreover, to mitigate the artifacts caused by the temporal inconsistency, they introduce a spatial-temporal decoder and fine-tuning strategy for the 3DGS optimization process. extensive experiments are conducted to demonstrate their superior reconstruction performance on large-scale datasets of unbounded scenes.

Strengths

1. Applying video models to scene reconstruction is a novel idea and has achieved excellent results. 2. To mitigate the negative impact of inconsistent generated images, this paper proposes improvements to the decoder and defines a confidence measure to enhance the corresponding 3D Gaussian Splatting optimization process.

Weaknesses

1. The writing of this paper is not very clear, and some typo errors exist in the submission, for instance, line147, it should be $ \{ I^{ref}_{i-1}, I_1, I_2, ..., I_T, I^{ref}_i \}$; line216: it should be $I_c$, consistent to the equation (8). 2. Training details are missing in the paper, more details about the video diffusion model and fine-tuning process should be provided. 3. More datasets, such as MipNerf360, should be evaluated to verify the effectiveness and robustness of the proposed method.

Questions

None

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

None

Summary

This paper is working on the problem of novel view synthesis with sparse input views. The authors present 3DGS-Enhancer to enhance the rendering quality and address 3D view consistency problem using 2D video diffusion priors. The experimental results show that this work has achieved state-of-the-art performance in novel view synthesis enhancement.

Strengths

1. Writing quality is great, clean, and easy to follow. 2. To tackle the sparse view reconstruction problem, the idea of fine-tuning the original 3DGS representation instead of incorporating additional geometric constraints is novel and promising. 3. A new dataset is introduced in this work, which is useful for this research direction and the community. 4. The idea of leveraging video LDM for sparse view reconstruction in unbounded outdoor scenes is novel and effective.

Weaknesses

Runtime is not discussed in this paper. I’m wondering how long it will take for one scene compared to the other baselines.

Questions

The experiments are conducted almost on the self-introduced dataset. Can 3DGS-Enhancer also surpasses the other baselines on more commonly used outdoor unbounded datasets (e.g. Mip-NeRF 360 dataset)?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

N/A

Reply to the authors

Thank you authors for your detailed reply and conducting the ablation study. I'm happy with your answers and the suggested changes; I shall be increasing my score.