RoboFlamingo-Plus: Fusion of Depth and RGB Perception with Vision-Language Models for Enhanced Robotic Manipulation

As robotics advances towards complex multimodal interactions and manipulation tasks, integrating Vision-Language Models (VLMs) has become crucial. Despite progress, challenges remain in fusing depth and RGB information in 3D environments and executing language-guided tasks. To address these, we introduce RoboFlamingo-Plus, enhancing RoboFlamingo by integrating depth data into VLMs, significantly improving manipulation performance. Our research achieves nuanced RGB-depth fusion by combining a pre-trained Vision Transformer (ViT) with resampling, aligning data with linguistic cues for superior multimodal understanding. RoboFlamingo-Plus adapts inputs for depth processing, uses a pre-trained resampler for depth feature extraction, and employs cross-attention for optimal feature integration. Experiments show RoboFlamingo-Plus boosts manipulation performance by 10-20%, marking a significant advancement.