MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry

We propose MAC-VO, a novel learning-based stereo visual odometry (VO) framework that trains a metrics-aware uncertainty model to serve two critical functions: selecting keypoints and weighting residuals in pose graph optimization. Unlike traditional geometric methods that favor texture-rich features like edges, our keypoint selector leverages this learned uncertainty model to eliminate low-quality features based on global inconsistency. In contrast to learning-based approaches that rely on scale-agnostic weight matrices for covariance, our metrics-aware covariance modelderived from the learned uncertaintycaptures spatial errors in keypoint registration and inter-axis correlations. By embedding this co-variance model into pose graph optimization, MAC-VO achieves superior robustness and accuracy in pose estimation, excelling in challenging environments with varying illumination, feature density, and motion patterns. Evaluations on public benchmark datasets demonstrate that MAC-VO surpasses existing VO algorithms and even some SLAM systems in difficult scenarios. Additionally, the uncertainty map offers valuable insights for decision-making.