Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completion

Current LiDAR-only 3D detection methods inevitably suffer from the sparsity\nof point clouds. Many multi-modal methods are proposed to alleviate this issue,\nwhile different representations of images and point clouds make it difficult to\nfuse them, resulting in suboptimal performance. In this paper, we present a\nnovel multi-modal framework SFD (Sparse Fuse Dense), which utilizes pseudo\npoint clouds generated from depth completion to tackle the issues mentioned\nabove. Different from prior works, we propose a new RoI fusion strategy 3D-GAF\n(3D Grid-wise Attentive Fusion) to make fuller use of information from\ndifferent types of point clouds. Specifically, 3D-GAF fuses 3D RoI features\nfrom the couple of point clouds in a grid-wise attentive way, which is more\nfine-grained and more precise. In addition, we propose a SynAugment\n(Synchronized Augmentation) to enable our multi-modal framework to utilize all\ndata augmentation approaches tailored to LiDAR-only methods. Lastly, we\ncustomize an effective and efficient feature extractor CPConv (Color Point\nConvolution) for pseudo point clouds. It can explore 2D image features and 3D\ngeometric features of pseudo point clouds simultaneously. Our method holds the\nhighest entry on the KITTI car 3D object detection leaderboard, demonstrating\nthe effectiveness of our SFD. Codes are available at\nhttps://github.com/LittlePey/SFD.\n