Deep Patch Visual Odometry

Summary

This paper proposes a deep learning-based method for monocular visual odometry, which equips two leading-edge advantages: (i) a deep feature-based patch representation for keypoints encoding local context, and (ii) a novel recurrent architecture designed for patches along with a differentiable bundle adjustment layer. The proposed DPVO outperforms the state-of-the-art across several common benchmarks while running 1.5-8.9x faster than the DROIOD-SLAM system with less memory leveraged.

Strengths

The proposed monocular visual odometry performs accurately in estimating the camera pose in several datasets. And the proposed system shows a strong generalization that is trained on the synthetic TarTanVO dataset but validated on real datasets.

Weaknesses

1. The paper makes claims that the proposed method outperforms all prior work in several evaluation datasets, but only several works are listed which is not enough, can authors give more comparisons with SOTA? For example, the DytanVO[1], RAM-VO[2], DF-VO[3] [1]https://arxiv.org/pdf/2209.08430v4.pdf [2] https://arxiv.org/pdf/2107.02974v1.pdf [3] https://arxiv.org/pdf/2103.00933v1.pdf 2. The results on various datasets are measured by ATE(Absolute Trajectory Error), but some other evaluation criteria are also essential, for example, the relative pose error (RPE), the average translational error, and the rotational error. Can the author provide more comparison information about them? 3. The tracking of patches is achieved by designing an RNN but fails to provide a clear explanation. Why not a CNN? 4. Punctuations are necessary at the end of each equation. Please carefully check it.

Questions

See Weakness part

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The paper claims that the current learning-based VO systems are impractical for resource-constrained devices. However, the proposed method still cannot be applied to mobile GPU-free devices.

Summary

The author propose a learning-based approach for visual odometry. It is essentially a sparse version of DROID-SLAM, without the loop closure capability. Instead of predicting the dense flow field relating the different frames as in DROID-SLAM, DVPO operates on sparse image patches, which are extracted at random from each frame. In more details, a random set of patches are extracted from each video frame, a patch graph is built. A recurrent neural network predicts the residual motion of each patch in each other frame. These residual motions are then used in a differentiable BA layer which minimizes the reprojection error to optimize for the sparse depth of each patches and the camera poses. This process is repeated iteratively until convergence. The paper proposes a new architecture to predict the residual motion, adapted to patches.

Strengths

1) Thorough and convincing evaluation: The proposed approach leads to a convincing improvement in terms of speed and accuracy, compared to DROID-SLAM. The approach is evaluated on many different datasets. 2) The ablation study also pin-points the essential contributions

Weaknesses

In DROID-SLAM, predicting the dense flow allowed to optimize for full dense depths along with the camera poses with accuracy, providing advantages over both direct and indirect approaches. However, the proposed approach DVPO is based on jointly estimating the motion of sparse patches and refining camera poses and sparse depth (of the patches). In that sense, it is not obvious what advantages are brought by coupling the matches estimation with the pose/depth prediction. Classical indirect approaches first predict sparse matches, triangulate 3d point to get the depth (thus providing initialization for the depth of ‘patches’), and then track further frame by performing PnP with multiple local and global BA. The paper compares to ORD-SLAM and outperforms it. However, there are now many sparse matching approaches that significantly outperform hand crafter detectors like ORB. How would DVPO compare to using a ‘classical’ indirect approach with matching performed by a deep-learning based state-of-the-art approach like SuperPoint-SuperGlue?

Questions

1) see weakness 2) A better figure for the patch graph would be help the understanding, showing what are the nodes, what are the edges, where the updates are and so on. Fig.1 is not very informative. It would significantly help to read sec. 3.1. I also find using the same index notation for the patches and for the images very confusing. ex L.159 (k, i) refers to an edge between patch k and image I. It would make it easier if the patches had a different notations like bold k. 3) some implementation details: how large are the patches? 4) in the ablation study, it is shown that using a patch is better than point feature. According to the formulation for estimating the motion of the patch, using a patch effectively increases the search window of the correlation in the other image. Does this difference still hold if the point feature is correlated to a larger window in the other image to estimate its motion?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

yes

Summary

In this paper, the author propose a new method to solve monocular visual odometry (VO) in and end-to-end fashion. In contrast to previous work - DROID-SLAM, which utilizes dense optical to build correspondences across frames, the proposed method leverages sparse patches to avoid redundancy of dense pixels and therefore works faster. Experiments on public datasets including TartanAir, TUM-RGBD, EuRoC and ICL-NUIM demonstrate its higher efficiency and competitive accuracy.

Strengths

1. Good motivation. Dense correspondences provided by optical flow estimation are not necessary for visual odometry because basically several good matches are enough to find a good pose (more matches may increase the robustness). Therefore, I agree with the author of using sparse patches to achieve higher time and memory efficiency. 2. Extensive results. The proposed method is evaluated on TartanAir, TUM-RGBD, EuRoC and ICL-NUIM datasets to show its efficiency and accuracy. 3. The paper is well-written and easy to read.

Weaknesses

1. Limited novelty. The key idea of the paper is relatively straightforward. If we review the classic monocular visual VO/SLAM systems, we can find most of them are based on sparse keypoints (ORB-SLAM) or semi-dense pixels (DSO) to avoid too much computation. From my point of view, the major contribution of the paper comes from the engineering part. The recurrent module designed to update the trajectories of patches is kind of novel as it automatically updates the covisibility graph constructed by observed patches and past frames. However, the overall novelty is still limited. 2. Related works. Another limitation is the discussion of previous works. In addition to DROID-SLAM, there are lots of excellent learning-based VO/SLAM systems proposed in the past a few years such as BeyondTracking [r1], NICE-SLAM [r2], iMAP [r3], and Li et al [r4] to name a few. These works have achieved SOTA performance. However, they are not discussed and compared neither. It would be better to give a discussion of these works in the paper. r1: Xue et al., Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry.CVPR 2019 r2: Zhu et al., NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. CVPR 2022 r3: Sucar et al., iMAP: Implicit mapping and positioning in real-time. ICCV 2021 r4: Li et al., DENSE RGB SLAM WITH NEURAL IMPLICIT MAPS. ICLR 2023.

Questions

1. L141: random sampling. It is not very clear that random sampling of patches works better. Intuitively, with the predicted camera motion, sampling according to the predicted pose could give more correct matches between patches. Even in the classic VO/SLAM systems like ORB-SLAM and DSO, they sample keypoints/pixels from regions with rich textures because these regions could be better observed in following frames. The comparisons of random sampling with other strategies could solidate the claim. 2. In the Differential Bundle Adjustment module, the cameras poses and inverse depths are optimized with pixel coordinates fixed. In practice, the matches between patches across frames have outliers, impairing the performance. Basically, the BA and outlier removing are jointly updated in classic systems to guarantee the accuracy. It seems that this can be achieved in the recurrent module and as this step is very important especially to monocular VO/SLAM systems, it would be great to show how the module ‘corrects’ wrong matches.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

This paper proposes a deep learning based visual odometry (VO) system which estimates the 6DoF poses of each frame and outputs a sparse reconstruction of the scene from an input monocular video sequence. The pipeline extracts deep features from the sampled image patches and updates deep optical flow of the patches with a recurrent module, then perform deep BA with patch correspondences within the local optimization windows on the fly. The reported statistics show that this system is able to run at 60FPS (Default) and 120FPS (Fast), respectively, while maintaining the camera pose estimation accuracy. Results show that the proposed method outperforms DROID-VO and ORBSLAM in terms of accuracy on TARTAN AIR, EuRoC MAV, and ICL-NUIM datasets.

Strengths

1. One of the main contribution of this work is to alleviate the large computational cost from the existing deep dense VO method (DROID-SLAM) while maintaining its pose estimation accuracy. 2. The paper shows the contribution of 1D Temporal Convolution and SoftMax Aggregation on neighboring frames to the pose estimation, which are novel. 3. The learned network is trained purely on synthetic data and it shows competency to perform well on the real datasets, including TUM-RGBD and EuRoC MAV. 4. The paper is clearly written and easy to follow.

Weaknesses

1. The novelty is limited, the iterative update mechanism and differential bundle adjustments are already used in DROID-SLAM, even with the introduction of 1D Temporal Convolution and SoftMax Aggregation modules in the update operator. 2. Another major contribution of the proposed method is to track image patches instead of using dense correspondence on the entire image. However, it is not theoretically sound to support the claim that the patch-based approach improves the accuracy of pose estimation results over the dense flow based approach when the patches are selected randomly rather than feature based methods. 3. I suspect that the model overfits the TartanAir dataset. That’s why there is a big performance gap between the results on synthetic and real datasets for the classic methods. When it comes to real datasets, the results are mixed. 4. The selection of baseline methods is inconsistent and questionable in Table 1-4, the ORB-SLAM version used is different, and authors did not compare the results to ORB-SLAM or D3VO on the EuRoC MAV dataset, while other VO papers do include them as baseline methods. Also, there are better results can be found for running ORB-SLAM on TUM-RGBD fr1 and TartanAir sequences (reported in https://paperswithcode.com/paper/droidslam-deep-visual-slam-for-monocular/review/).

Questions

1. Are there any insights for the reasons of randomly sampled patches outperforming other feature based patch selection mechanisms (ORB, SIFT, and Superpoint)? Although the empirical results in this paper favor randomly sampled patches, it is still counter-intuitive to imagine most patches are randomly sampled on regions with repetitive patterns or occlusion. Are there any additional experiments to demonstrate the robustness of random patch selection? 2. As mentioned in Weaknesses, why ORB-SLAM or D3VO are not included when comparing the results on the EuRoC MAV dataset? Are these methods not considered visual odometry systems? 3. Both DROID-SLAM and DVPO are trained from synthetic datasets, are there any suggestions on the reasons for the worse performance of DROIDSLAM than DPVO on the ICL-NUIM dataset, even with global bundle adjustment? 4. Are the running parameters (window size, patch size, number of update iterations) of the system tuned for different datasets, or are they fixed? Does patch size matter (not including the 1x1 point feature)? 5. Are there any comparison between the system and classical methods in terms of efficiency, with different input frame sizes?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The model can only be trained on synthetic datasets with ground truth optical flow. Although the system shows the ability of generalization over the real data, its performance and robustness may be limited in the real scenarios.

rebuttal

Thanks for the responses. The newly added comparisons with previous VO methods (r1, r2, r3, r4) make the results of the proposed method more convincing. I don’t have any more questions about the experiments and hope the analysis and discussions of these results can be included in the refined manuscript. My major concern, however, is still the novelty. As claimed by the author in the rebuttal and also mentioned by the reviewer Uwmu, the major contribution is to use randomly sampled patches to replace dense optical flow in DROID-SLAM to achieve higher efficiency. I don’t see more novelty from this design in spite of the better performance which could be achieved from the engineering.