DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization

Summary

This paper proposed a dynamic RGB-D SLAM system based on 3D Gaussian Splatting (3D-GS) and DROID-SLAM. Semantic segmentation masks and depth warping residuals are used to generate motion masks to remove the dynamic part of the scene. Experiments on three real-world datasets show the proposed approach achieves superior results on both dynamic and static scenes.

Strengths

1. The paper is well written, technically sound and easy to follow. 2. The idea of using depth warping residuals as a complement to semantic priors for motion mask generation is simple but works effectively well. 3. The proposed method is very well engineered and achieved promising results on real-world dynamic sequences. 4. The proposed method is the first dynamic SLAM method based on 3D Gaussian Splatting (3DGS).

Weaknesses

1. The technical novelty is a bit limited. The paper focuses on adding new components to previous SLAM methods in order to achieve dynamic SLAM. 2. The hybrid coarse-to-fine camera tracking seems like a trick: The coarse stage directly comes from DROID-SLAM while the fine stage simply adds masks to 3DGS-based tracking.

Questions

Please see Weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper introduces the first robust dynamic visual SLAM system based on 3D Gaussian Splatting. It provides precise camera pose estimation and high-fidelity reconstructions. Strategies such as motion mask generation, adaptive Gaussian point management, and hybrid camera tracking are proposed to improve the accuracy and robustness of pose estimation.

Strengths

The writing is clear and easy to understand. The paper proposed the first GS-based SLAM systems for dynamic scenes. Compared with previous NeRF-based and GS-based SLAM systems, it achieves optimal performance on dynamic scenes.

Weaknesses

1. The evaluation conducted in the paper primarily compares the proposed method against NeRF or Gaussian-splatting-based SLAM systems, which is not sufficient. It would be beneficial to include traditional SLAM systems as baselines, which present a more robust tracking capability, such as ReFusion [1] and ORB-SLAM3 [2]. 2. Lacks the comparison with MonoGS [3] as the baseline of GS-based SLAM. MonoGS demonstrates more advanced map reconstruction ability compared with other GS-based alternatives, and presents robust camera tracking utilizing Jacobian optimization (e.g. compared with trivial pose optimization via photometric loss implemented in SplaTAM). It should be included as a baseline method for more exhausted comparisons. 3. Lacks survey on the literature of related work, such as Crowd-SLAM [4] and DDN-SLAM [5], which were also proposed for the same motivation of reconstruction on the dynamic environment. 4. The experiments on two datasets have shown the effectiveness (see point 5) of the proposed method in dynamic indoor settings. However, for a SLAM system, it would be preferable to conduct experiments on real-world scenes, especially in outdoor environments. 5. The proposed system largely relies on dynamic object removal in the tracking system; however, it lacks certain optimization for the mapping system. This results in the reconstructed scenes still containing artifacts. [1] Palazzolo, J. Behley, P. Lottes, P. Giguere, and C. Stachniss, “Refusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals,” in IROS, 2019. [2] C. Campos, R. Elvira, J. J. G. Rodr ́ıguez, J. M. Montiel, and J. D. Tard ́os, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” IEEE TRO, 2021. [3] Matsuki, Hidenobu, et al. "Gaussian splatting slam." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [4] Soares, João Carlos Virgolino, Marcelo Gattass, and Marco Antonio Meggiolaro. "Crowd-SLAM: visual SLAM towards crowded environments using object detection." Journal of Intelligent & Robotic Systems 102.2 (2021): 50. [5] Li, Mingrui, et al. "Ddn-slam: Real-time dense dynamic neural implicit slam with joint semantic encoding." arXiv preprint arXiv:2401.01545 (2024).

Questions

1. According to Table 6, the segmentation mask seems to play a much more significant role than other components, such as the depth warp mask. Please discuss the utility of the other components in detail. A concern here is that a more accurate segmentation mask (e.g. produced by SAM) might facilitate even better dynamic removal without the need for other components. 2. In the implementation detail, the author should elaborate more on how to perform semantic segmentation, the parameter of depth warping, and the keyframe selection strategy they have integrated in the proposed system. 3. The dynamic features are removed in the tracking system; however, a large portion of the removed features might reduce the tracking accuracy. In other words, to what extent can the method proposed in the paper tolerate motion? Is there any failure case/example of tracking, e.g. when dynamic features dominate the current view? 4. The semantic mask is retrieved by utilizing a pretrained segmentation model. There is a concern regarding the generalizability of this semantic mask. How is the dynamic object defined with respect to the semantic mask? For instance, the human might be considered a dynamic object in the BONN dataset, but might be stable in other scenes (e.g. wax figure). In addition, how does this integration of the semantic model affect the real-time processing capability of the SLAM system?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

There is no societal impact of this work, and limitations are discussed in the paper.

Summary

The paper present DG_SLAM to address the inconsistent observation of geometry and photometry in dynamic environments among the current slam-related field. Specifically, the authors develop several technologies such as the motion mask generation, adaptive gaussian point management, and a hybrid camera tracking algorithm to improve the performance of slam system. And the experiments on TUM RGB_D and BONN RGB_D dataset showcases the state-of-the-art performance both on static and dynamic environments.

Strengths

1. For the task of motion object segmentation, the authors propose an advanced motion task generation method which integrates spatio-temporal consistent depth masks combined with semantic priors, and several rigorous mathematic formulations are illustrated to demonstrate the effectiveness. 2. A coarse-to-fine camera pose optimization is proposed to optimize the whole slam system, and improve the constancy between pose and reconstructed 3d map. 3. The authors also present an adaptive gaussian point addition and pruning strategy to manage the generated gaussian map.

Weaknesses

1. A motion mask generation strategy is used to mitigate the potential impact of observation noise from a single warp mask, can the authors give more details about this part? Are there any theoretically analysis for that? 2. A hand-crafted rule is adapted in the map point deleting stage. For example, the authors delete points based on three baselines: the opacity value, the maximum scale, the ration between ellipsoid’s major and minor axes, can authors give more explanation about these criteria? Moreover, the authors showcase the removement of number of observations of the point that are too low, but fail to show its quantitative criteria. 3. the running time on BONN dataset during the mapping phase seems a bit large in table 7, can authors give several details about this result? Are there any connections with the proposed map point deleting strategy?

Questions

see weakness part

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The method fails to be conducted on several large-scale real scenes.

Summary

The paper combines existing methods for deep SLAM (DROID-SLAM) and 3D Gaussian splatting for 3D mapping into a system and adds an approach for dynamics filtering by depth warping to make tracking more robust in dynamic scenes. The approach is evaluated on several dynamic SLAM benchmarks and compared with recent deep learning based RGB-D SLAM approaches.

Strengths

- The proposed approach of dynamics filtering for SLAM with 3D Gaussian splatting for map representation seems novel. - Experiments demonstrate improvements over previous methods for neural RGB-D SLAM which have been designed for static scenes.

Weaknesses

- Unfortunately, this paper has significant shortcomings in writing quality and clarity. Many variables and concepts are not defined properly (for instance, J, R in eq 2., \otimes operator, I_{m\times n}, \Pi in eq 8, S_i in l. 192, the symbols in eq. 13). - Some phrases in the introduction are not accurate or proper English. What does "finishing pose estimation robustness" mean? What are invalid zones, what does furnishing an initial pose estimate mean? - Sec. 2, the first paragraph starts with the heading "traditional visual SLAM with dynamic objects filter", but contains recent deep SLAM approaches for static scenes. Please restructure. - The raycasted color and depth in eq. 3 requires a specific ordering of the gaussian splats, as indicated in the products ranging from j=1 to i-1. Please explain how it is established. - What does the intersection operation over pixels achieve in eq 6 and 9? What does it mean to apply an indicator function and take a \otimes product with the I_mxn ? What is I_mxn? - The variable \mu is used for coordinates and means of 3D Gaussians ambiguously, please change notations. - l. 179, how is the camera pose parametrized and optimized ? - l. 188, what is the dynamic radius? - l. 192, what is the scale vector S_i and why should it be computed from the mean distance of three nearest neighbor points? Where do these points come from? - eq 12, how are the lambda_1-3 chosen? - l. 235, "unit quaternion algorithm" is not the right name for the method but rather Horn's procrustes method. - l. 257, what is an "analytical experiment" ? - Table 2, 3, what are the metrics for the numbers ? ATE, RPE ? how is it computed? - Tables 2, and 3 should also compare with classical dynamic SLAM methods like Co-Fusion [*1], MID-Fusion [*2], EM-Fusion [*3] to name a few. Also these methods are missing from the discussion of related work. - what is an "invalid Gaussian point filter" in l. 272 ? - Table 5 seems to indicate that different settings of the iteration count hyperparameter are needed for the various datasets. This indicates an issue with generalization for the method. Are other hyperparameters chosen consistently across datasets? - l. 290 what does to "encompass major motion categories" mean ? - l. 291, what is the definition of the STD metric ? - Table 7 / l. 302, please include run-time for semantic segmentation as it seems essential according to Table 6. - How is the semantic segmentation mask obtained ? What is the runtime of this processing step? - The paper does not discuss limitations and assumptions of the method beyond not performing loop closing. [*1] Martin Ruenz, Lourdes Agapito. Co-fusion: Real-time segmentation, tracking and fusion of multiple objects. ICRA 2017 [*2] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew J. Davison, Stefan Leutenegger. MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM. ICRA 2019 [*3] Michael Strecke, Joerg Stueckler. EM-Fusion: Dynamic Object-Level SLAM With Probabilistic Data Association. ICCV 2019

Questions

See questions in paper weaknesses.

Rating

3

Confidence

5

Soundness

2

Presentation

1

Contribution

2

Limitations

The conclusion provides an obvious limitation of the approach due to not considering loop closing. Further limitations and assumptions are not discussed.