CP-SLAM: Collaborative Neural Point-based SLAM System

Summary

The paper presents a collaborative implicit neural simultaneous localization and mapping (SLAM) system for RGB-D image sequences, which is able to merge sub-maps into a global map and also handle loop-closure for neural mapper for the first time. To facilitate the capability of doing loop-closure and sub-map merging, the authors introduce a neural point-based 3D scene representation, where each point maintains a learnable neural feature for scene encoding and is associated with a keyframe. The optimization follows the traditional SLAM’s graph optimization fashion. The experimental results on diverse datasets demonstrate the superior performance of the proposed method in both camera tracking and mapping.

Strengths

1. Novel Neural Point-based 3D Scene Representation: A neural point cloud representation for scene encoding in the proposed framework is a significant contribution. This approach allows each point to carry encoded information, improving the system's ability to represent and cross-view association for large-scale tracking and pose refinement. 2.Comprehensive Collaborative Framework: The authors provide a collaborative SLAM system with front-end and back-end modules, addressing essential aspects such as odometry, loop detection, sub-map fusion, and global refinement. This comprehensive approach ensures that the proposed system can handle various challenges typically encountered in SLAM tasks. 3.Distributed-to-Centralized Learning Strategy: The incorporation of a distributed-to-centralized learning strategy is an effective solution for improving consistency and cooperation in collaborative implicit SLAM. 4.Improved Performance: The experimental results on multiple datasets demonstrate the superiority of the proposed method in both camera tracking and mapping.

Weaknesses

1.In the Introduction: This paper emphasizes the significance of targeting collaborative-SLAM, considering the substantial value already provided by loop-closure capability in effectively reducing pose drift. However, the evaluation of the paper becomes more challenging due to the necessity for additional comparative studies that can convincingly demonstrate the advantages offered by both approaches. 2.In Section 2: For collaborative mapping, this paper should cite more recent works including: [1*] Campos, Carlos, et al. "Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam." IEEE Transactions on Robotics 37.6 (2021): 1874-1890. [2*] Tian, Yulun, et al. "Kimera-multi: Robust, distributed, dense metric-semantic slam for multi-robot systems." IEEE Transactions on Robotics 38.4 (2022). It is also expected to provide comparative studies to the above researches. 3.In Section 3: (1). Line 101: This paper introduced 4x4 patches to compute the point features. However, it misses details on there is a limitation on the number of patches. What if we increase the size of the patches? Will it impact the later performance? (2) This paper introduced so many symbols but without giving any explanation of the symbols, which makes it really hard to follow the paper. Please revise the whole paper and make them clear to the readers. (3) Equation 10: The explanation “Considering the strong non-convexity of the color map, only the geometry part is included in tracking loss” for not using color in the loss is not convincing. The authors are expected to provide ablation studies to prove the claim. 4.In Section 4: More experiments are expected: (1). More comparative studies on Collaborative-SLAM are expected. (2). For single-agent cases: The local covisibility map can be formed as well; it would be interesting to only look at cases without a loop but having a local covisibility map. (3). Ablation study on “Distributed-to-Centralized Learning” is needed to prove why this is needed.

Questions

1.In the context of this paper, which relies on point features for establishing frame-to-frame correspondence, an important concern arises regarding the maintenance of continuous tracking. Unlike the traditional SLAM pipeline, where feature locations typically reside around corners, this paper selects points without necessarily adhering to this characteristic. Thus, it becomes crucial to address how this approach ensures the continuity of tracking throughout the SLAM process. 2.Collaborative mapping can be done by recent work on No Pose Nerf: []Bian, Wenjing, et al. "Nope-nerf: Optimising neural radiance field with no pose prior." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. This paper should provide a comparison to show the advantage of the proposed method. 3. This paper has too many contributions, which does not form a scientific paper but more an engineering paper. Not sure if this paper fall into the NeuralIPS group.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. The writing should be improved, especially the equations should be better explained. 2. More experiments and ablation studies should be provided to demonstrate the advantage of the proposed algorithms. 3. The literature review needs to refer to more recent research.

Summary

This paper proposed a collaborative RGB-D neural implicit SLAM method named CP-SLAM. CP-SLAM is based on neural point representation similar to PointNeRF, which is natually suitable for collaborative SLAM and loop closing. Based on this neural point representation, CP-SLAM is able to perform full SLAM with a single or multiple agents: tracking, mapping, loop detection, loop closing, submap merging, etc. Experiments show that CP-SLAM achieves SOTA results compared with previous collaborative SLAM and neural implicit SLAM methods.

Strengths

1. Using neural point representation in the context of neural implicit SLAM is novel and the motivation of using it is also clear, i.e. more suitable for loop closing and sub-map merging. 2. A full SLAM system that is capable of loop closing, Pose Graph Optimisation and sub-map merging. This is the first time I saw a full SLAM system in the context of neural implicit SLAM. 3. The system seems well engineered, putting many components together and works effectively well. 4. Competitive tracking and mapping results compared with previous collaborative and neural SLAM method.

Weaknesses

The motivation is clear and the method is very well engineered, but from novelty perspective, all the components in CP-SLAM are already there with some incremental modification: e.g. the scene representation is from PointNeRF, Loop detection is from an off-the-shelf NetVLAD model, etc.

Questions

1. From Sec. 3.1 it seems that at each frame a new set of neural points are created densely via back-projection, and there doesn't seem to be any point merging scheme. Then how is CP-SLAM able to achieve 0.62MB parameters size in Tab. 4? 2. Also from Sec. 3.1 it seems that the rendering loss will only optimize the neural features stored in the points but will never affect the locations of the points, which means once created the location of a point will never change unless a loop is detected and a Pose Graph Optimisation is performed. If this is the case, then the entire mapping process would be equivalent to simply back-projecting the depth images? 3. How did you convert your point-based map to mesh for evaluation in Tab. 3? 4. Following my question in 2: if CP-SLAM's map points are obtained explicitly from depth measurement, then it is supposed to have better accuracy than NICE-SLAM and Vox-Fusion (the same reason that TSDF-Fusion has better accuracy than neural SLAM methods). Thus, better accuracy numbers in Tab. 3 is not enough to prove CP-SLAM have "more powerful geometric reconstruction capability". Also it's not clear why the authors didn't report completion. After rebuttal: the authors have addressed all my questions and concerns.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes

Summary

The authors propose a novel collaborative RGB-D SLAM pipeline, where the map is represented by an implicit field based on Point-NERF that are transformed together with their keyframes allowing pose graph optimization. They use a loop closure detection based on learned features (NetVLAD) also for sub-map alignment. They propose a central federated learning based averaging and refinement of the distributed MLPs after submap alignment.

Strengths

The use of point based features nicely connects the generalization and compression of implicit mapping with the ability to do efficient map refinement. The results on tracking and reconstruction show a large improvement compared to existing works.

Weaknesses

The explanations are sometimes a bit vague. E.g. line 101 what does it mean to incrementally project a pixel to a 3d location? The initialization of system is not described well. While the authors describe the averaging and finetuning of the MLP weights, but do not discuss if and how the point features are shared between agents. Evaluation was only done on synthetic data.

Questions

Why is there another local finetuning and weight averaging done after the central averaging and finetuning? How is the finetuning done after pose graph optimization? Is it the same as for mapping e.g. eq 9? Would it not be possible to run ORB-SLAM3 (https://arxiv.org/pdf/2007.11898.pdf) in a multi-map setting using depth from RGB-D as a baseline? Why not evaluate on real data? How is the TSDF fusion done to generate the final mesh? What is the average baseline between frames in the dataset? How long are the trajectories? After rebuttal: The authors addressed all of my questions. Thank you.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

One of the main limitations of framerate/view point change is addressed by the authors.. an initial experiment to showcase the effect would have been nice though.

Summary

The paper introduces a neural implicit representation based collaborative SLAM technique. They introduce a learnable feature encoding for each point in the 3d scene based on neural rendering subsequently using it for depth and color map estimation. They introduce a centralized learning strategy to share MLPs and learn better weights. They also introduce a LM algorithm based global pose optimization and key frame centric point cloud adjustment.

Strengths

The federated learning strategy improves the neural rendering performance. Map refinement based on keyframe centric neural point field. By encoding scene geometry and appearance with 3D points, when camera poses are optimized, these features are easily adjusted as well.

Weaknesses

1. The paper only evaluates on one synthetic datasets. It would be nice to see evaluations on more datasets 2. Better and more robust neural rendering techniques could have been used to improve the radiance, depth and color rendering

Questions

1. The paper is not clear whether it includes neural field generation time in tracking/mapping time while comparing with Nice SLAM. 2. The paper lacks citations. For example, in the mapping and tracking section, a similar technique is used in Nice SLAM and I see no mention of it.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1.The paper does produce SOTA results on some of the scenes of the dataset chosen. No explanation is given to why it doesn't produce better results on the rest. 2. The paper is not novel enough. It rather combines the best of many other works and produces good results.

Summary

The paper presents CP-SLAM, a collaborative neural implicit SLAM system developed by the authors. CP-SLAM incorporates neural point-based odometry, loop detection, sub-map fusion, and global refinement. Additionally, the authors propose a novel neural point 3D scene representation with keyframes to enhance map fusion and adjustment. They introduce innovative learning and optimization frameworks to ensure accurate and consistent 3D mapping for cooperative localization and mapping. The performance of CP-SLAM is evaluated on various indoor RGB-D sequences, showcasing its superior mapping and camera tracking capabilities. The contributions made by the authors significantly advance the field of SLAM and offer state-of-the-art solutions.

Strengths

1. CP-SLAM introduces a novel neural point-based representation, enabling dense collaborative neural SLAM. 2. CP-SLAM retains the front-end and back-end modules like traditional SLAM systems, providing a comprehensive pipeline. 3. CP-SLAM outperforms state-of-the-art methods in both localization and reconstruction. The CP-SLAM system supports both single-agent and multi-agent modes, allowing for versatile evaluation. 4. The system is evaluated using datasets based on the Replica scenes, providing a realistic and representative testing environment. CP-SLAM is compared against recent neural implicit RGB-D SLAM methods in single-agent experiments, showcasing its performance in loop closure. 5. In the two-agent experiments, CP-SLAM is compared with a traditional method, highlighting its collaborative capabilities. 6. The ablation studies are conducted to demonstrate the importance of different modules in the proposed system. 7. The implementation details, such as GPU specifications and parameter settings, are provided, ensuring reproducibility.

Weaknesses

1. CP-SLAM requires significant GPU resources to handle multiple image sequences. The relative pose computation in loop closure relies on existing rendering-based optimization, which may be inaccurate for large viewpoint changes and lead to map fusion drifting. 2. The comparison with traditional methods in the two-agent experiments may not fully represent the performance of other collaborative neural SLAM approaches. 3. CP-SLAM's performance is limited by its GPU resource demand, making it less suitable for resource-constrained devices or environments. 4. Currently, a large-scale experiment is not conducted, but for multi-agents, this is an important point to consider.

Questions

1. How does CP-SLAM handle scalability in terms of the size and complexity of the datasets? 2. Can CP-SLAM be applied to real-world datasets to evaluate its performance in practical scenarios? 3. Are there any specific limitations or challenges in the implementation of CP-SLAM that need to be addressed? 4. Can Nicer-SLAM and DROID-SLAM be compared as a contrast? 5. The article mentioned "Naturally, it is expected that the neural point cloud layout should be rearranged following global pose graph optimization. However, a world-centric point cloud map obviously cannot allow such adjustment. To tackle this limitation, we propose a keyframe-centric neural point field, where each 3D point is associated with a keyframe". Can the authors better explain why a world-centric point cloud map obviously cannot allow such adjustment? Sincerely,

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations have been discussed in the conclusion section.

Summary

There have been some papers on getting dense Visual SLAM with neural fields published in the last couple of years. In this work, the authors proposed a set of additions to prior pipelines that get the NN SLAM closer to the traditional approaches concerning the overall framework. Specifically, the authors addressed loop detection, pose graph optimization, bundle adjustment, and collaborative SLAM.

Strengths

1) Visual-SLAM with neural fields has the huge benefit of providing a dense map of the environment. The topic is relevant to computer vision, robotics, and I believe it fits the NeurIPS scope. 2) The paper is well-written, easy to follow, and well-organized. Related work is OK. 3) Although the technical novelties are relatively limited, in my opinion, the contributions to the pipeline are significant.

Weaknesses

1) As written above, the technical contributions are limited. The paper focuses more on adding new steps to prior Visual SLAM pipelines. 2) The inclusion of the NetVLAD (not new) as a tool for loop detection looks like a trick. It is not elegant, in the sense that it is a parallel feature extractor that (maybe) could be done using the neural field encoders. 3) Only one dataset and 4 sequences are used, which is small concerning quantity and variety. 4) I would prefer to see more comparisons with previous traditional methods (without neural fields). 5) Figures 5 and 6 are not easily readable.

Questions

1) I would like the authors to explain what is the motivation for the inclusion of NetVLAD. Can't those descriptors be obtained from the neural field? 2) It is written that a pre-trained NetVLAD was used. It would be important to give more details about this training and how it generalizes to different scenes. 3) Usually, with classical V-SLAM, the bundle adjustment estimates both the pose of the cameras and the 3D map simultaneously. It seems that the authors are doing this alternatively. I would like to see some comments on this. 4) Why are the results for Apartment 2 so bad for the proposed approach? I would like the authors to comment on this. 5) It seems that traditional V-SLAM methods using RGB-D are missing (such as OrbSLAM). I would like to see some comments on this. 6) Maybe I missed something. But it is not clear to me how is the error in Tab. 2 computed.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are adequately addressed.

Comment

Dear authors, thank you for your responses and added analyses. We appreciate that the authors add the comparisons against DROID-SLAM and the comparisons in the global response. These analyses should be added to the final version. We would like to maintain the positive rating and suggest that the authors make the code publicly available to foster future research in collaborative SLAM. Sincerely,

2nd round comments

Thank you to the authors for their responses. The authors largely addressed my last comments. But there are still several more questions: 1. After reading the responses, the reviewer still believes that the authors should separate it into two papers: 'Point-SLAM' and 'Collaborative Mapping.' Alternatively, the authors have to highlight the motivation in the Abstract and Introduction. Otherwise, it is really hard to understand why the authors want to include Collaborative Mapping in this paper. 2. The authors answered, 'From the perspective of accuracy, we empirically found that an increased quantity of neural points slightly decreases the tracking and mapping performance due to the increased complexity and redundancy of the neural point field.' This contradicts the conventional feature-based tracking SLAM approach. This will make the proposed method difficult to generalize and use. I hope the authors can address my concerns and I would like to increase my rating.

More evaluations and limited novelty

The authors have included more datasets and experiments which achieve SOTA performance. The authors have also corrected the issues with paper writing. Though the authors have developed and achieved a good pipeline for the whole collaborative NERF-based SLAM system, I think the novelty is still a bit limited.