MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting

Summary

The proposed method extends the idea of GaussianFlow to explicitly decouple camera motion and object motion from optical flow on input monocular video. Specifically, the iterative camera pose refinement further boosts the rendering quality and performance on various datasets. Extensive qualitative and quantitative results are shown in the paper to valid the proposed contributions.

Strengths

1. The paper extends the idea of GaussianFlow to explicitly decouple the camera motion and object motion from optical flow, which is reasonable and the formulation of the proposed method also makes sense. 2. Explicitly modeling camera motions in an iterative way during training is useful for monocular NVS because the rendering error partially brought by incorrect camera pose estimation could be further decreased. Though the camera pose refinement is not originally introduced by the paper, the decoupling scheme is highligh. 3. Ablation study (both the main paper and appendix) is detailed and inspiring, validating the main contributions claimed in the paper. 4. Both qualitative and quantitative results look good.

Weaknesses

No prominent weakness is witnessed.

Questions

1. Since the paper showcased the failure case in Fig.13, I would like to know whether the results could be improved if the camera refinement step was removed (leaving that optical flow equals to motion flow) considering camera poses in DyNeRF dataset are fixed? (Original GaussianFlow paper minimized the difference between optical flow and Gaussian flow, where the proposed method minimized the difference between Gaussian flow and motion flow, which is optical flow - camera flow given camera flow in DyNeRF is 0).

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations have already been included and acknowledged in the paper. The proposed method seems to be less effective on cases with less camera motions.

Summary

This paper proposes using off-the-shelf 2D optical flow to supervise the deformation field for 3D Gaussian Splatting (3DGS) in dynamic scenes. The optical flow is decomposed into camera flow and motion flow. The 3DGS flow is projected into 2D to match the estimated flow. Camera pose and Gaussian parameters are optimized iteratively. Experiments demonstrate the effectiveness of the proposed method.

Strengths

1.The formulation and utilization of optical flow are reasonable, and the experiments demonstrate its effectiveness.

Weaknesses

1.The idea of using optical flow to guide dynamic 3DGS modeling is not particularly novel; it seems more like an engineering effort. 2.With the additional 2D optical flow supervision, the improvement shown in the experiments is subtle.

Questions

1.Without explicitly modeling reflections, how does it perform well on the NeRF-DS dataset, even outperforming NeRF-DS with additional modeling? 2.With additional optimizations, how much longer would the training time be compared to the baseline and other methods?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, other limitations are listed in the weakness.

Summary

The paper proposed MotionGS a novel deformable 3D Gaussian splatting approach. The approach initializes camera poses and 3D Gaussians based on an analytic structure-from-motion method as 3DGS. In addition an optical flow network is used to compute optical flow between neighboring frames. Given the initial depth of the 3D Gaussians, the paper proposes to use an optical flow decoupling module separating the camera and object motion. This process is optimized to obtain better camera poses and better views through 3D Gaussian splatting and then iterated. The paper uses Gaussian flow which has been very recently proposed by Gao et al., 2024, however only matching the flow due to object motion. The iterative improvement of the camera poses is based on the estimation of small residuals between consecutive views. The paper reports experiments on the NeRF-DS and the HyperNeRF dataset and achieving high construction quality and improving of Deformable-3DGS. An ablation study is presented comparing object motion flow with optical flow without separation and the effects of camera pose refinement. The supplemental material contains more details on Gaussian flow, more implementation details, visualizations and discussions. The appendix also contains a further ablation study quantifying the influence of the depth, optical flow etc.

Strengths

Explicit motion guidance for 3D Gaussian reconstruction is a logical next step for deformable 3D Gaussians. The paper combines and extends some very recent parallel work in optimization of camera pose estimation by Fu et al., 2024 and optical flow based Gaussian flow by Gao et al., 2024. The paper demonstrates the successful application of the proposed MotionGS framework to non-rigid scenes recorded by video streams. The reported reconstruction quality of MotionGS is competitive and often state-of-the-art for videos in the chosen datasets.

Weaknesses

The paper has a large range of limiting assumptions which are not always made clear. COLMAP must find an initial solution for camera pose estimation which requires enough static features in the scene. A video input stream is needed to ensure that neighboring viewpoints are very close for optical flow estimation to succeed. The 3DGS canonical representation must be found in order for depth estimation to work. The datasets used in evaluating the proposed method can be processed with deformable 3DGS and hence the proposed MotionGS method is limited to improving reconstruction quality. No reconstruction beyond the limitation of prior work is demonstrated and the above limiting assumptions makes this likely challenging. The quality of camera pose estimation is not well evaluated. There is only a visualization in the appendix which is very hard to interpret. (See Fu et al. 2024 for a more meaningful evaluation). The proposed method appears to be quite brittle as can see from the ablation study in Table 5 in appendix A.3. The method falls below the baseline by leaving out the motion mask, switching to single image depth estimation and even by switching to another SOTA flow estimation network.

Questions

The key for MotionGS appears to be the separation of optical flow into object flow and camera flow. The paper does not really review work in this area, would there other ways to attribute the flow to object and camera motion? If Deformable 3DGS fails to find a reasonable canonical representation at initialization, is there a fallback to start the iterative optimization? On which data was the optical flow network trained on, and has the optical flow network been finetuned on the data? Would self-supervised optical flow methods which are known to generalize better, improve the flow estimation?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors provide a limitation statement in Appendix A.5 but focused only on optical flow failures. The claim of optical flow failure for static cameras is surprising, maybe a shortcoming of the particular flow estimation. It would be more interesting to investigate the robustness to large motions which likely needs to severe flow failures. In addition the limitation of structure-from-motion and depth estimation are not discussed.

Summary

This paper presents a novel approach to dynamic scene reconstruction by incorporating explicit motion priors into 3D Gaussian Splatting (3DGS). The proposed framework, MotionGS, introduces an optical flow decoupling module that separates camera flow and motion flow, which respectively correspond to camera movement and object motion. This separation allows for more precise motion guidance during the deformation of 3D Gaussians. Additionally, a camera pose refinement module is implemented to alternately optimize 3D Gaussians and camera poses, addressing inaccuracies in camera pose estimation. Extensive experiments demonstrate that MotionGS outperforms state-of-the-art methods on datasets such as NeRF-DS and HyperNeRF, achieving significant improvements in both qualitative and quantitative results for dynamic scene reconstruction.

Strengths

1. The paper is written clearly: The authors present a well-structured flow from the problem definition, through the intuition behind the approach, to the implementation and analysis. Each section is closely connected to the main point, making the paper easy to follow. 2. The proposed method is novel, intuitive, and simple to implement: MotionGS introduces a unique approach by decoupling optical flow into camera flow and motion flow, providing explicit motion guidance for 3D Gaussian Splatting. This method is innovative and straightforward, making it easy to adopt and integrate into existing systems. 3. Experiments validate the results well, with extensive visualizations and ablation studies: The paper robustly validates the proposed method with extensive experiments on datasets like NeRF-DS and HyperNeRF. Numerous visualizations and detailed ablation studies clearly demonstrate the effectiveness of each component of MotionGS.

Weaknesses

1. The time, memory, and storage costs are not revealed: While the proposed method is illustrated as a simple solution, it is crucial to analyze the additional computational burden it imposes. The paper does not provide an analysis of the time, memory, and storage requirements, which are important factors in evaluating the practicality of the method. 2. The data sampling mechanism is missing: The paper lacks details on the data sampling mechanism used for training. It is unclear how image pairs are sampled—whether t and t+1 can be any two frames in the video or they must be adjacent to each other. If the former is true, it would imply that N(N−1) optical flow maps need to be calculated, which could significantly impact storage and time requirements. Providing these details is essential to understand the overall efficiency and feasibility of the approach.

Questions

If no flow net is adopted, is it possible to calculate the loss as follows? 1. Estimate optical flow as the sum of camera flow and motion flow 2. Using optical flow to warp I_t+1, and calculated the loss with I_t.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No negative societal impact

Good job

I am satisfied with the response and I will keep my positive score.

Detailed Rebuttal

I thank the authors for their detailed rebuttal with additional experiments, figures and tables. The performance drop with self-supervised MDFlow is very interesting. In general, I find the answers informative and helpful. My additional comments are as follows: I think, it would be helpful to readers (especially those less familiar with 3DGS) to include the discussion in Section 0.1 into the paper. I find the explanation of the reduced performance with monocular depth estimation both interesting and very reasonable. Ideally, one could evaluate improvement of camera poses with synthetic data where ground truth is known but I acknowledge the difficulty as real-world challenges of using COLMAP must be included at the same time. Maybe, this can be left to future work. In my experience, it is not uncommon for SfM to completely fail in very dynamic real-world scenes. I appreciate that imprecise pose estimates can be improved, however, depending on the imagery complete failure is a challenge. Again maybe something for future work.

Replying to Rebuttal by Authors

Thanks for clarification about the contribution: camera and object motion sepation from the estimated optical flow, especially for the cases with complex motion and inaccurate poses. Given the additional analysis and training time comparison, I would like to raise the score.

Issues addressed well

All my concerns have been addressed well. Besides, I appreciate that the authors have evaluated the self-supervised setting. I adjust my score to 7 and suggest putting this extra try on the supplementary material.