FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes

Summary

This paper introduces a generalizable 3DGS model which is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. The key idea of this paper is introducing Low-cost Cross-View Aggregation, which makes it is possible to use more nearby views for feature matching. Experiments show the advance in long sequence novel view synthesis.

Strengths

1. Low-cost Cross-View Aggregation is introduced to predict initial Gaussian triplets, with which the computational cost would be lower, making it possible for feature matching between more nearby views and training on long sequence reconstruction. 2. The Gaussian triplets are fused by Pixel-wise Triplet Fusion. This module can effectively reduce the pixel-aligned Gaussian redundancy in the overlapping regions and aggregate multi-view 3D Gaussian latent features. 3. The experiments show the advance in long sequence novel view synthesis.

Weaknesses

1. It would be better if the authors could report the training time of their model, since the authors addressed that both the Low-cost Cross-View Aggregation module and Pixel-wise Triplet Fusion module can lower the computation cost. 2. It would be better to include the comparison with SurfelNeRF, since using a depth estimator differs from the GT depth. 3. The definition of FreeSplat-fv and FreeSplat-spec is not given clearly in this paper.

Questions

1. I want to make it clear that FreeSplat-fv indicates using Free-View Training (FVT) strategy and FreeSplat-spec indicates using specific number of reference views, such as two or three. 2. Gaussian Splatting SLAM is one of the 3DGS-based SLAM system, which also can be run without depth as input information.It would be better if the authors can explain the strength of their work compared with this kind of 3DGS-based RGB SLAM, since the SLAM method can run in real time and process a longer input sequence.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes. As the authors discussed in this paper, the GPU requirement becomes expensive (> 40GB) when inputting extremely long image sequence (> 50). Besides, there is still a gap between the method of this paper and the other methods that rely on depth information as input.

Summary

This paper proposes a FreeSplat, aiming at generalizable 3D gaussian splitting for long sequence inputs. Specifically, it uses an efficient CNN-based cost volume and eliminates redundant 3D gaussians observed across multiple views. Extensive experiments show that FreeSplat effectively reduces inference costs and improves the novel view quality for long sequence inputs.

Strengths

* The writing is good, making it easy to understand and follow. * The proposed Pixel-Aligned Triplet Fusion (PTF) is interesting as it effectively handles the redundant 3D gaussians for long sequence inputs.

Weaknesses

* I think the evaluation details are not clear. For example, during the evaluation, I am curious whether the maximum gap between input views is fixed regardless of the number of input views. Does "view range" increase as the number of input views increases? If so, please describe the evaluation details. * It is weird that the novel view image quality drops as the number of input views increases, but the novel view depth quality increases. I'd like more clarification on this. * The existing pixelSplat and MVSplat have been experimented with on RealEstate10k [1] and ACID [2], and both datasets also contain long sequence inputs. The authors should have included comparisons on at least one of the RealEstate10k and ACID datasets. [1] Zhou et al., Stereo magnification: Learning view synthesis using multiplane images, SIGGRAPH 2018. [2] Liu et al., Infinite nature: Perpetual view generation of natural scenes from a single image, ICCV 2021.

Questions

The proposed method is interesting, but the lack of evaluation details, lack of comparative experiments, and inconsistencies in the results make it difficult to verify the contribution. The authors should address the "weaknesses" and strengthen the manuscript.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

1. Low-cost Cross-View Aggregation: This efficient methodology constructs adaptive cost volumes between proximate views and aggregates features utilizing a multi-scale structure. This approach enables the processing of extended input sequences and the incorporation of more stringent geometric constraints. 2. Pixel-wise Triplet Fusion (PTF): This module aligns and merges local Gaussian triplets into global representations, thereby mitigating redundancy in overlapping regions and consolidating features observed across multiple viewpoints. 3. Free-View Training (FVT): This novel training strategy decouples the model's performance from a predetermined number of input views, thus enabling robust view synthesis across an expanded range of viewpoints. 4. The authors demonstrate that FreeSplat surpasses existing methodologies in both novel view synthesis quality and depth map accuracy on the ScanNet and Replica datasets. The proposed method exhibits enhanced performance in view interpolation and extrapolation tasks, particularly for extended input sequences. Moreover, FreeSplat reduces the quantity of redundant Gaussians and offers more computationally efficient inference compared to preceding approaches.

Strengths

1. The paper is well-structured, offering lucid explanations of the technical approach. It includes detailed architectural diagrams and pseudocode to enhance reproducibility. Comprehensive ablation studies are presented to isolate the impact of different components. The overall writing is coherent and accessible. 2. This research enables free-viewpoint rendering and 3D reconstruction from arbitrary numbers of input views, broadening the applicability of 3D Gaussian splatting techniques. It enhances efficiency and minimizes redundancy in 3D Gaussian representations for extended sequences.

Weaknesses

1. Experimental Scope and Comparability: PixelSplat presents experimental results on the Re10k and ACID datasets. For a comprehensive and equitable comparison, it would be beneficial for the proposed method to also report results on these two datasets. This would facilitate a direct comparison with pixelSplat across multiple benchmarks, thereby strengthening the validity of the authors' claims. The absence of experimental results on Re10k and ACID raises questions about the proposed method's performance and generalizability across diverse datasets. I am wondering why the experimental results on Re10k and ACID are absent. 2. Analysis of Failure Cases and Limitations: The paper would be significantly enhanced by a more thorough examination of failure cases and limitations of the proposed approach. While the appendix briefly touches on this aspect, a more in-depth analysis would provide valuable insights into the method's robustness and potential areas for improvement. Such an analysis could include specific examples of scenarios where the method underperforms, a discussion of the underlying causes for these failures, and potential strategies for addressing these limitations in future work. This level of critical self-evaluation would not only increase the paper's scientific rigor but also provide a more balanced perspective on the method's capabilities and constraints.

Questions

Please see the weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors discuss some limitations of their work: 1. High GPU memory requirements (>40GB) for extremely long input sequences (>50 images). 2. A gap in 3D reconstruction accuracy compared to state-of-the-art methods that use 3D supervision or RGB-D inputs.

Summary

This paper proposes FreeSplat to reconstruct geometrically consistent 3D scenes from long sequence inputs. To this end, the paper presents Low-cost Cross-View Aggregation for feature matching and Pixel-wise Triplet Fusion for Gaussian triplets fusion. The outstanding results of long sequence 3DGS generalization shows the superiority beyond baseline methods.

Strengths

1. The paper is well-written and the technical contributions are clearly clarified and easy to understand. 2. The experimental results over baselines are attractive and convincing, which demonstrates the effectiveness of the proposed method. 3. The proposed pixel-wise triplet fusion is reasonable and novel.

Weaknesses

1. The low-cost cross-view aggregation module is not novel enough. It seems that the three submodules (2D feature extraction, cost volume formulation, multi-scale feature aggregation) are all borrowed from existing methods and the low-cost cross-view aggregation is a simple stack of these submodules. I suggest the author to further clarify the differences between the proposed module and the existing MVS methods.

Questions

1. What are the differences between Freeplat-spec and Freesplat-fv? 2. The author provided 3-view version results in the long sequence reconstruction experiment in Table 2. I’m doubting why do you provide the 10-view version of the baselines? Just because their original paper did not provide the 10-view version, or because they will fail to reconstruct a scene under 10-view inputs, or other reasons? Further clarification will help to understand this experiment. 3. The author claims that the proposed pixel-wise triplet fusion module helps to remove redundant Gaussians in the overlapping regions and the experimental results of final Gaussian number demonstrates this. However, it is somewhat not intuitive. It would be better to provide a visualization to show how the Gaussian ellipsoids are reduced in the overlapping regions.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The author discussed the limitations in the appendix about the memory consumption and the limited performance of generalizable Gaussians, which I think is valuable. It remains a significant problem for further study.

About Figure 2

Thanks for your rebuttal! Sorry for the typo in my review. "I’m doubting why do you provide the 10-view version of the baselines?" should be "I’m doubting why don't you provide the 10-view version of the baselines?", but your rebuttal has solved my question. I am still unclear about Figure 2 in the rebuttal PDF. Do you mean that all of the Gaussians in the blue mask are fused and removed? Why are there still some excluded Gaussians in the second image? Is it because they are retained after the fusion process? Could you give some more clarification on this?