MambaPlace:Text-to-Point-Cloud Cross-Modal Place Recognition with Attention Mamba Mechanisms

Vision-Language Place Recognition (VLPR) enhances robot localization performance by incorporating natural language descriptions from images. By utilizing language information, VLPR directs robot place matching, overcoming the constraint of solely depending on vision. However, general multimodal information integration methods are not well equipped to capture the dynamics of cross-modal interactions, especially in the presence of complex intra-modal and inter-modal correlations. To this end, this paper proposes a novel coarse-to-fine and end-to-end connected cross-modal place recognition framework, called MambaPlace. In the coarse-localization stage, the text description and 3D point cloud are encoded by the pre-trained T5 and instance encoder, respectively. They are then processed using Text-Attention Mamba (TAM) and Point Cloud Multi-Strategy Scanning Mamba (MSSM), with the latter mimicking the eye’s focusing mechanism, for data enhancement and alignment. In the subsequent fine-localization stage, the features of the text description and 3D point cloud are cross-modally fused and further enhanced through Cascaded Cross-Attention Mamba (CCAM). Finally, we predict the positional offset from the fused text-point cloud features, achieving the most accurate localization. Extensive experiments show that MambaPlace achieves improved localization accuracy on the KITTI360Pose dataset compared to the state-of-the-art methods. Specifically, as shown in Fig. 1, when ϵ<5, MambaPlace achieves 5% higher test accuracy compared to the existing state-of-the-art.