Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes

Summary

This paper introduces a new unsupervised approach and a new network architecture for monocular depth estimation. The paper is motivated by the necessity of new algorithms capable of learning to predict depth from unlabelled data while being able to perform reliable predictions in the regions of moving objects, which is particularly critical for autonomous driving applications. To tackle this problem, the authors proposed a new architecture that separates the observable flow field as rigid regions and independently moving regions. A training scheme is also proposed to effectively learn this disentanglement guided by a motion masking that is also learned in a self-supervised manner. The proposed approach is evaluated on three datasets and achieves state of the art results on nuScenes and Waymo Open, specifically improving by 50% on moving objects.

Strengths

The main insight in the proposed method is to decompose the flow field used to obtain depth supervision as rigid flow F_R, which goal is to explain the flow field due to the camera motion as if the scene where rigid, and independent flow F_I, which models the flow field resulting from moving objects. Decomposing these two factors is not trivial and the paper proposes an interesting solution to the problem that does not require annotated labels. The insights presented about the later overfitting of depth estimation models to dynamically moving objects is a very interesting finding and supports the main idea in the paper. The authors successfully explored this observation to propose a new training scheme for learning the motion mask without requiring segmentation labels. The results are convincing both quantitatively and qualitatively, as provided in the supplementary material, specially for moving objects. The paper is well written and relatively easy to follow.

Weaknesses

As in many previous works, the proposed approach targets the depth estimation problem as a novel-view synthesis problem. The particular case of moving objects is then tackled by a separated 3D flow field that captures the independent motion of moving objects, which is coherently discussed in the paper. Although this setup seems to be generic, the proposed approach is only evaluated in the context of autonomous driving. Therefore, it is not evidenced in the paper if the proposed formulation would adapt well to different contexts if a moving camera. For instance, if the camera is moving in a way that results in no parallax effect, could this framework diverge to a solution with most of the pixels being segmented as moving objects? The main contribution of the paper is in the network architecture and specially in the training strategy. However, the idea of braking the scene rigidity assumption by estimating a motion field for moving objects was already explored in [11] and the notion of sparse moving objects in the scene as a prior was also explored in [17]. The main differences with these two works in special are only briefly mentioned in the paper.

Questions

How the ground plane d^g_t used in Eq. 13 is defined? The authors mentioned that it is estimated via RANSAC, but is this assuming that the ground is mostly empty and visible? What happens in traffic jam situations? Please clarify this part of the method and provide what are the assumptions for the ground plane estimation. Typos: L48 "learning erroneous depth for the moving objects and allow." L51 " achieve achieve" L198: F_D instead of F_d ?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Limitations were briefly discussed in the paper and no major concerns are remaining.

Summary

This paper aims to estimate the depth map in a dynamic scene environment. To this end, the paper proposed 1) two separate module architectures that estimate rigid scene flow and residual scene flow and 2) a motion initialization method to enable a stable learning process. However, the technical and performance comparisons with the previous self-supervised depth estimation, self-supervised scene flow estimation, self-supervised motion segmentation, and self-supervised optical flow are insufficient. The current manuscript needs a lot of modification.

Strengths

- The proposed method achieves better performance in nuScenes and Waymo datasets. However, the comparison group is too limited to judge the superiority of the proposed method.

Weaknesses

W1. Technical and performance comparisons with the recent self-supervised scene flow estimation The proposed method needs to describe its originality and superiority compared to recent scene flow estimation on the KITTI, nuScene, or Waymo datasets. - Xiang, Xuezhi, et al. "Self-supervised learning of scene flow with occlusion handling through feature masking." Pattern Recognition 139 (2023): 109487. - Jiao, Yang, Trac D. Tran, and Guangming Shi. "Effiscene: Efficient per-pixel rigidity inference for unsupervised joint learning of optical flow, depth, camera pose and motion segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. - Hur, Junhwa, and Stefan Roth. "Self-supervised monocular scene flow estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. - Hur, Junhwa, and Stefan Roth. "Self-supervised multi-frame monocular scene flow." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. - Guizilini, Vitor, et al. "Learning optical flow, depth, and scene flow without real-world labels." IEEE Robotics and Automation Letters 7.2 (2022): 3491-3498. W2. Technical and performance comparisons with the recent self-supervised motion segmentation The proposed method needs to describe its originality and superiority compared to recent motion segmentation on the KITTI, nuScene, or Waymo datasets. - Liu, Liang, et al. "Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity." IJCAI. 2019. - Jiao, Yang, Trac D. Tran, and Guangming Shi. "Effiscene: Efficient per-pixel rigidity inference for unsupervised joint learning of optical flow, depth, camera pose and motion segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. - Xiang, Xuezhi, et al. "Self-supervised learning of scene flow with occlusion handling through feature masking." Pattern Recognition 139 (2023): 109487. W3. Technical and performance comparisons with the recent self-supervised optical flow estimation The proposed method needs to describe its originality and superiority compared to recent optical flow estimation on the KITTI, nuScene, or Waymo datasets. - Teed, Zachary, and Jia Deng. "Raft: Recurrent all-pairs field transforms for optical flow." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020. - Jonschkowski, Rico, et al. "What matters in unsupervised optical flow." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020. - Zhao, Wang, et al. "Towards better generalization: Joint depth-pose learning without posenet." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. W4. Technical and performance comparisons with the recent self-supervised depth estimation The proposed method needs to describe its originality and superiority compared to recent self-supervised depth estimation on the KITTI, nuScene, or Waymo datasets. - Watson, Jamie, et al. "The temporal opportunist: Self-supervised multi-frame monocular depth." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. - PLADE-Net: Towards Pixel-Level Accuracy for Self-Supervised Single-View Depth Estimation with Neural Positional Encoding and Distilled Matting Loss, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. - Guizilini, Vitor, et al. "Multi-frame self-supervised depth with transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Questions

My major questions are shown in the weakness part.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Limitations

This paper must include in-depth technical and performance comparisons with the recent self-supervised depth, scene flow, optical flow, and motion segmentation literature. The current manuscript doesn't support the proposed method's originality or superiority without the comparisons.

Summary

This paper proposes a self-supervised depth estimation method. It addresses dynamic objects in the scene by jointly learning monocular depth, 3D independent flow field, and motion segmentation from unlabeled videos. The paper writing is very good, and the motivation/insights are clearly stated. The method is evaluated in KITTI, Waymo, and nuSecens datasets. Overall, I think that it is a high-quality paper.

Strengths

The paper is well-written, and it is easy to read. The results are evaluated in Waymo and nuSecens datasets. This is very good because the KITTI dataset has been overfitted, and it contains fewer moving objects than the other two datasets. The Sec 3 is very informative. The discussion on motivations makes the paper easy to understand. The evaluation results and ablation studies clearly demonstrate the efficacy of the proposed method.

Weaknesses

There is no obvious weakness to me, but I think that it would be better if the authors could show visual odometry results. The nuSences and Waymo datasets provide camera poses, so it is not hard to do. [a] also proposes a method for dynamic objects, but it uses a different solution. The comparison is not required, but it would be better if making a discussion on different ideas for addressing dynamic objects. [a] SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes, arXiv 2022

Questions

NA

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

NA

Summary

This paper presents a method for monocular depth estimation in dynamic scenes, employing an unsupervised learning approach. While prior work utilizes networks for depth estimation, pose estimation, flow estimation, and mask estimation to decompose object motion and agent motion, the proposed method in this paper similarly adopts these network structures. Distinguishing itself from earlier works, this paper introduces an initial estimation of motion segmentation for the joint learning of depth and independent motion. The main contribution lies in providing an effective initial estimation of motion segmentation, which is critical for determining which pixels in a scene are moving independently. An ablation study demonstrates the effectiveness of the proposed initialization technique. Furthermore, the method achieves state-of-the-art performance on the Waymo and nuScenes datasets.

Strengths

- The proposed method significantly outperforms the state-of-the-art in the Waymo and nuScenes datasets. - The inclusion of an ablation study effectively illustrates the significance of the proposed initialization techniques. - The paper introduces a simple yet effective technique that enhances the performance of depth estimation. - Comprehensive training details are provided, along with evaluations concerning various regions, including static backgrounds, static movable objects, and moving objects.

Weaknesses

- The proposed method fails to surpass the performance of existing approaches on the KITTI dataset. - The paper primarily focuses on the initialization technique for motion segmentation, which appears overly simplistic and may limit the scope of the paper.

Questions

- Is the proposed network trained separately for each dataset (KITTI, Waymo, nuScenes), resulting in distinct sets of parameters as displayed in Table 1? Alternatively, is there a single trained network that is tested across all three datasets? - The 'Model and Training setup' subsection appears to describe a training schedule. Is this training schedule uniformly applied across all datasets (KITTI, Waymo, nuScenes), or is it adapted according to specific dataset properties such as size and resolution?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Authors adequately addressed the limitations.

Summary

This paper deals with monocular depth estimation by optical flow estimation. The key is to decompose the flow as ego-motion-induced flow and independent flow. To implement the idea, depth is predicted on the target image, and camera pose change from target to source is also estimated. Then a flow network predicts the flow from the target to the source, and a motion mask is also predicted. Then the final flow that warps the source to the target is recomposed by an alpha combination of the independent flow from the flow network and the rigid flow computed from the depth and the pose change. Experiments are performed on Kitti, nuScenes, and Waymo Open.

Strengths

Explicit modeling of the ego-motion of the ego car and the independent motion of the moving car can help improve depth estimation. Since the disentanglement can help optimize the right part of the parameters, otherwise, if there is no explicit modeling, then the network can tradeoff the accuracy on the depth to get an accurate optical flow due to ambiguities.

Weaknesses

The idea of modeling independent motion and ego motion for depth estimation, flow estimation, and moving object estimation is not quite new. For example, this idea has appeared in the following papers: - "UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos" CVPR 2019. - "GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose" CVPR 2018 - "Towards scale-aware, robust, and generalizable unsupervised monocular depth estimation by integrating IMU motion dynamics" ECCV 2022 - "Dyna-DM: Dynamic Object-aware Self-supervised Monocular Depth Maps," 2023 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC) - "Learning Optical Flow, Depth, and Scene Flow Without Real-World Labels" ICRA 2023 - "Instance-Aware Multi-Object Self-Supervision for Monocular Depth Prediction" RAL 2022, seems to have better results on Kitti.

Questions

1. "Prima facie, the epipolar ambiguity should not be that much of a problem for a learning-based depth estimation system. This is because independent object motion should be generally inconsistent between different videos, and the depth estimator network cannot recover this information from a single frame alone. Thus, in a hypothetical world filled by moving spheres, the motion information is unobtainable from a single frame and the depth estimation network would indeed regress to the correct depth." can you explain a bit what is the underlying reasoning that the ambiguity should not be a problem? The writing is not explicit, and I have to guess the meaning of words, e.g., "the depth estimator network cannot recover this information" what does "this information" mean? why if the motion is unbotainable then the depth estimation network would indeed regress to the correct depth? 2. please clarify the major contribution of this work.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors claimed no societal impacts.

Thank you for the rebuttal

After carefully reading the rebuttal, I appreciate the clarifications provided. As a result, I have decided to adjust my initial rating upwards. While the main idea of this paper is straightforward, its effectiveness across various works is commendable. Furthermore, the authors adequately address the challenges faced in surpassing the existing state-of-the-art results on KITTI.