SCube: Instant Large-Scale Scene Reconstruction using VoxSplats

Summary

This paper presents a method to reconstruct large-scale 3D scenes from a sparse set of posed images. The method has two stages: the first stage is for voxel grid reconstruction, which is based on the XCube [38], a 3D generative model, and the authors introduce the image condition to the model. The second stage is for appearance reconstruction, which is a feedforward appearance prediction model to predict a certain number of Gaussians for each voxel. The authors also propose a GAN-based post-processing step to boost the appearance of the reconstructed scene. The proposed method is evaluated on the Waymo self-driving dataset, showing the superiority of the proposed method compared to prior art on 3D reconstruction tasks. Besides, the authors also demonstrate the applications of the proposed method, such as LiDAR simulation and text-to-scene generation.

Strengths

- The proposed method addresses an important problem in 3D vision, i.e., reconstructing large-scale 3D scenes from a sparse set of posed images. The authors cleverly take advantage of the pre-trained 3D diffusion generative model to relieve the ill-posedness of the problem. - Although technically the proposed method is not entirely new, the paper presents a framework to combine the voxel grid reconstruction and appearance prediction, and the presented result looks promising to me. - The paper is well-written and easy to follow. The authors provide a clear explanation of the proposed method and the experiments.

Weaknesses

- The proposed representation's abbreviation "VoxSplats" is somewhat confusing to me, since the term "splatting" in original 3DGS refers to the rendering technique that splats the 3D Gaussians onto the image plane, rather than the 3D representation. The proposed representation is more like a hybrid of voxel grids and Gaussian representation. The authors may consider using a more accurate term to describe the representation. - It's unclear how the foreground and background are separated in the proposed method. Using a panorama to represent the sky makes sense as the sky is usually far away from the scene, but some buildings are treated as the background and seem to be represented by the panorama as well in the supplementary video. A more detailed explanation and discussion of the foreground-background separation is needed. - There are some lighting/exposure/white-balance inconsistencies in the reconstructed scenes in the supplementary video. The authors should discuss the limitations of the proposed method in handling the lighting/exposure/white-balance variations in the input images. - For SCube+, how to guarantee consistency among the rendered views? Also, rendering with the proposed GAN-based post-processing step is time-consuming, which would destroy the valuable real-time rendering performance of the Gaussian Splatting. The authors should discuss these issues in the paper. - Missing the evaluation of the geometric accuracy of the reconstructed scenes. As the rendering quality of SCube is not so good, geometric accuracy is crucial to the applications of the reconstructed scenes. The authors should provide a detailed evaluation of the geometric accuracy of the reconstructed scenes.

Questions

The most important questions that I would like the authors to address are: - Geometric accuracy evaluation of the reconstructed scenes.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes, the authors have discussed the limitations of the proposed method.

Summary

- The paper proposes a new method for sparse-view 3D reconstruction using 3DGS. - The framework uses two stages: 1) it learns a latent voxel grid (based on XCube) to represent the geometry. 2) it trains an appearance model to decode the latent voxel grid into a set of Gaussians - The method further uses a background model to handle the sky. - The authors propose an (optional) GAN postprocessing which improves the visual quality at the cost of longer (20m) per-scene optimization. - The method was evaluated on the Waymo Open Dataset, and it outperforms compared methods.

Strengths

- The paper is very well written and easy to follow. - The method is novel and the results look good. - I really like the idea of having probabilistic latent space representation (VAE). - The paper contains an ablation study illustrating the tradeoff between model complexity and performance as various hyperparameters are changed.

Weaknesses

- Section Method "Training and Inference" is not very clear. It would be preferable to expand the diffusion loss from Appendix A, and clearly describe the input/targets. - I would also like to have more details on the Sky Panorama model. - The method was evaluated on a single dataset. It would be interesting to see how the method performs on other datasets.

Questions

- How does the method compare to other methods like MVSplatting, MVSGaussians? How does it differ? How do you think it would compare in terms of performance?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

- Limitations are properly discussed in the paper.

Summary

The paper introduces a pipeline for street scene reconstruction given a sparse set of images as input. The reconstruction process follows a feed-forward manner. The method builds upon the existing XCube work. First, it generates sparse voxels of the scene to represent the geometries, then each voxel feature is decoded into Gaussian primitives for appearance rendering. Experimental results, both quantitative and qualitative, show that SCube can reconstruct the scene with high quality and provides benefits for downstream applications such as autonomous driving simulation.

Strengths

(1) The results are impressive. Given the sparse images with little overlap, the reconstructed Gaussians are of high quality and show good generalization ability in novel views. (2) Overall, the method sounds reasonable. Technical details are provided, and I believe the results are reproducible.

Weaknesses

(1) I am curious if we really need a diffusion model for this task. Since there are several images as conditions, the uncertainty (or randomness) of the output should be very small. Why not just train a regressor for the sparse voxel reconstruction? Is there any specific motivation for using a diffusion model?

Questions

(1) Why is the GAN loss used independently for each scene in the inference stage, instead of using it in the training stage?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Currently, the results are presented in the sparse view settings of a small region of the street scene. How can we reconstruct a very large scene using a long sequence of images in a feed-forward manner? This is an interesting topic for future work.

Summary

The paper proposes a new method to reconstruct 3D outdoor scenes from a sparse set of posed images. The key idea is to utilize a new hybrid 3D representation, which assigns 3D Gaussians to each sparse voxel. Given input images, the paper first adapts XCube to condition on sparse view images. Along with the generated fine-level voxels, the paper then designs a feed-forward model to reconstruct the appearance. This proposed pipeline can also be used in text-to-scene generation. Experiments show that the proposed method outperforms existing approaches.

Strengths

Originality: The paper is trying to solve a challenging problem: efficient large-scale scene reconstruction from sparse posed images. The motivation is to utilize the data prior to complement the sparse images. The proposed representation is a straightforward combination of XCube with 3D Gaussians (simply assigning 3D Gaussians into each voxel), which has also been widely applied (e.g., Scaffold-GS). While combining these two representations is acceptable, the concern is that the overall pipeline design seems more like a simple combination of two representations: first generating XCube, then generating 3D Gaussians with geometry & images. However, the problem to be solved is important, and this paper might inspire more researchers to work on this problem. Quality: The submission seems technically sound. Clarity: The paper is mostly clear and easy to read. Significance: Though there are some concerns with the methods and results, the paper is aiming at solving a very good problem.

Weaknesses

- One concern is that the novelties of this paper are not very clear. As stated above (Originality), the paper seems to simply use two stages to combine the two representations in a simple way: one for XCube, and one for 3D Gaussians. It would be better if the author can clearly state the contributions. - As a two-stage method, it would be better to discuss more about: (1) First stage: how to evaluate whether the reconstructed voxels align with the input images? (2) Second stage: how robust is the model when the input images and the voxels are not aligned? - L187: Considering the temporal inputs, can the methods give consistent results temporally? Adding results might be better. - L244-245: In Table 1, SCube seems to have very similar results to PixelSplat for future prediction. Why? An obviously better reconstruction result alone seems not able to show the effectiveness of the proposed method. The appearance reconstruction uses generated geometry (fine-level voxels) as input, and adds some tricks such as sky modeling. It is unknown how they affect the results.

Questions

See Weaknesses

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A