Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

Following a navigation instruction such as 'Walk down the stairs and stop at\nthe brown sofa' requires embodied AI agents to ground scene elements referenced\nvia language (e.g. 'stairs') to visual content in the environment (pixels\ncorresponding to 'stairs').\n We ask the following question -- can we leverage abundant 'disembodied'\nweb-scraped vision-and-language corpora (e.g. Conceptual Captions) to learn\nvisual groundings (what do 'stairs' look like?) that improve performance on a\nrelatively data-starved embodied perception task (Vision-and-Language\nNavigation)? Specifically, we develop VLN-BERT, a visiolinguistic\ntransformer-based model for scoring the compatibility between an instruction\n('...stop at the brown sofa') and a sequence of panoramic RGB images captured\nby the agent. We demonstrate that pretraining VLN-BERT on image-text pairs from\nthe web before fine-tuning on embodied path-instruction data significantly\nimproves performance on VLN -- outperforming the prior state-of-the-art in the\nfully-observed setting by 4 absolute percentage points on success rate.\nAblations of our pretraining curriculum show each stage to be impactful -- with\ntheir combination resulting in further positive synergistic effects.\n