Memorize What Matters: Emergent Scene Decomposition from Multitraverse

Summary

This paper presents a 3D Gaussian mapping framework that is able to convert multitraverse videos from the same region into a environment while segmenting out 2D ephemeral objects. Leveraging the mlutitraverse data, the scene decomposition emerges in an unsupervised manner, as a result of the consensus in background and dsisensus in transient foreground across traversals. The paper demonstrate promising results in object segmentation, 3D reconstruction, and neural rendering.

Strengths

+ The paper is overall well written and easy to follow. + The proposed approach is self-supervised, which does not rely on any object annotations, and hence holds good potential for scalability. + The paper demonstrates the value of multitraverse data in scene decomposition, which is relatively under exploited in prior work.

Weaknesses

- The method performs segmentation and finally GS mapping sequentially, which means the error in segmentation would propagate into the subsequent GS mapping. Why not perform multiple rounds of segmentation and GS mapping to continue improving the robustness against outliers? - The novelty of the paper is somewhat limited. Similar idea of emergent scene decomposition has been demonstrated in prior work EmerNeRF, where DINO features are also used. This paper differs mostly in using multitraverse data, and adopting 3DGS as scene representation instead of NeRF. - In object segmentation task, the paper compare with unsupervised segmentation method, but lacks comparisons with object discovery methods. As noted in Sec. 6, the proposed method is highly relevant to object discovery. - The comparison with STEGO and CAUSE is in a sense unfair because they do not use multitraverse data. A simple baseline would be performing a matching (e.g. with SIFT) for each segmented region with other traversals, a failure of which indicates the region being transient and vice versa. - For 3D environment reconstruction, only depthanything is compaired, which is however a single-image depth estimation, whereas the proposed method is multiview based. How about comparing with SOTA multiview stereo approaches? In addition, why is Chamfer Distance adopted as evaluation metric? Chamfer distance is a weaker metric as it does not leverage correspondence to the ground truth, but here we do have pixel-level correspondence by projecting Lidar to the image plane. Lastly, is the depth evaluation carried out only on background regions? There is no explanation on this regard. - As shown in Table 3, the neural rendering performance of EnvGS is only marginally better than the original 3DGS. It is even lower than 3DGS in PSNR.

Questions

The questions to be answered are detailed in the weakness section, mainly on stronger baselines in object segmentation, and evaluation on 3D mapping.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The paper has discussed the limitations.

Summary

The paper presents a novel approach for self-supervised scene decomposition using multi-traverse camera data, which results in a high-quality static background scene reconstruction via Gaussian Splatting. The method 3D Gaussian Mapping leverages repeated traversals and feature distillation to capture the emergent focus on 2D consensus structures and, therefore, dynamic foreground masks, which contribute to mitigating the disturbance of temporal dynamic objects on visual-based 3D mapping. Along with the proposed benchmark, this method paves an interesting direction, leveraging traversals to learn the inherent structure of background in autonomous driving, which finds wide applications in 3D map construction, map change detection, driving simulation, etc.

Strengths

1. **Innovative Approach**: The combination of multitraversal and Gaussian Splatting is a fresh and compelling method. It effectively decomposes an urban scene into static and dynamic elements merely via visual features and image rendering, which is an innovative idea for the urban scene decomposition task. 2. **Method of Simplicity and Effectiveness**: The paper combines and adapts the latest feature distillation, denoised DINOv2, and 3D reconstruction methods, Gaussian Splatting, to accomplish self-supervised dynamic component segmentation through emergent outweighing effect by multitraversal data. Visual feature residuals resulting from rendering results are effectively utilized to extract contours of the ephemeral elements simply by spatial gradient with the following postprocessing. 3. **Comprehensive Evaluation and Ablation Study**: The evaluation section is thorough, demonstrating the method's effectiveness across three tasks, i.e., 2D segmentation, 3D reconstruction, and 3D mapping. The abundant qualitative and quantitative results show robust performance under diverse conditions and improvements over existing techniques, particularly in ephemerality decomposition and 3D scene geometry learning. An encompassing ablation study shows the influence of traversals and selection of hyperparameters on segmentation tasks.

Weaknesses

1. **Assumptions on Environmental Stability**: The method strictly assumes a stable environment without major geometry change under consistent illumination and weather. This assumption might not hold in wider real-world scenarios, potentially affecting the method's robustness and generalizability. 2. **Lack of Failure Case Study**: Even though the paper discusses some failure cases regarding shadow, occlusion, reflection, etc. It's still interesting to see the influence of weather and large illumination changes on this approach and if increasing the number of traversals can tackle the issue. 3. **Lack of Comparative Baselines Regarding Env Reconstruction and Rendering**: Although the paper provides a solid evaluation of all tasks, it could benefit from more comparative analysis with state-of-the-art urban scene reconstruction methods, especially regarding static environment representation.

Questions

1. **Additional Comparison to Urban Scene Reconstruction Methods**: Some comparisons to the latest urban scene reconstruction methods, like EmerNeRF, HUGS, NeuRad, and so on, regarding the static env reconstruction and rendering quality, especially to the results of actor removal. This can serve as a broader discussion on the effect of multitraversals and sensor types. 2. **Failure Case Study**: Despite the strict assumption of this approach, it's still compelling to discuss the effect of environment and illumination change on this approach. 3. **More Details on Training and Rendering Procedure** This paper offers a detailed workflow as an overview of the whole approach. However, directly embedding Dino features in 3DGS can have side effects on training and rendering, especially on this large scene. More details on hyperparameter selection and the training process can be helpful for further work in this field. Overall, this paper presents a promising and innovative advancement in scene decomposition from multitraverse data, with several strengths that make it a valuable contribution to the field of computer vision. Addressing the noted weaknesses could further enhance its impact and generality.

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

NA

Summary

The paper proposes a method called 3DGM that performs foreground-background disentangled 3D reconstruction by capturing the consistent parts from multi-traverse videos. 3DGM leverages 3DGS as the scene reconstruction algorithm, using only camera images as input, and achieves decoupled reconstruction of the 3D environment and 2D object segmentation. Additionally, the paper introduces a new dataset that combines Ithaca365 and nuPlan, which is used to evaluate unsupervised 2D segmentation, 3D reconstruction, and neural rendering. To be specific, the author has observed that self-driving cars often traverse the same routes repeatedly, encountering new pedestrians and vehicles each time, similar to how humans encounter different people every day in the same 3D environment. Inspired by the fact that humans are better at remembering permanent structures and forgetting transient objects, the author proposes the idea of developing a mapping system that can identify and memorize the consistent structures of the 3D world through multiple traversals without the need for human supervision. The key insight is that while the specific pedestrians and vehicles change from one traversal to the next, the underlying 3D environment (buildings, roads, etc.) remains the same. By leveraging this observation, the proposed system could learn to filter out the transient objects and focus on mapping the permanent structures, akin to how humans naturally encode spatial knowledge. The goal is to create a self-supervised system that can build a robust 3D map of the environment simply by repeatedly navigating through it without requiring any manual labeling or annotations.

Strengths

3DGM is an unsupervised approach that doesn't require any additional manual annotations. The reconstruction process relies solely on camera input, without the need for depth measurements from sensors like LiDAR. Employing 3DGS enables much faster reconstruction speeds for implicit neural rendering. The authors also contribute a new multi-traverse video dataset.

Weaknesses

The end-to-end process for 3DGM appears to be quite complex, with multiple stages involved. However, the paper fails to provide a lucid explanation of the nitty-gritty details involved in implementing the algorithms. Without the availability of the accompanying source code, it would be an uphill task for others to replicate the results or build upon this work effectively.

Questions

Please check the Weaknesses and Limitations

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Firstly, although this is a self-supervised approach, in practical applications, multiple passes along the same road are still required for reconstruction. Could some prior information be incorporated to enable the model to infer the background and complete the task with fewer traversals? Secondly, the methods based on 3D reconstruction have high data quality requirements, and the 3DGS-based methods need a reasonably accurate initial pose estimation (by COLMAP), which further exacerbates the demand for high-quality data.

To reviewers

Dear reviewers, This paper received mixed ratings, the author(s) have provided a rebuttal. It would be appreciated if reviewers could carefully read the rebuttal and let the authors and the AC know if it has changed your mind (and detailed justification). Your opinion will play a significant role in the final decision of this paper. Thank you so much. Best Regards, AC

Thanks for the response

Hi, I appreciate the detailed responses to my concerns. Thanks for agreeing with me in many aspects. I am happy to stick on my initial rate.