SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency

In this paper, we explore how we can build upon the data and models of\nInternet images and use them to adapt to robot vision without requiring any\nextra labels. We present a framework called Self-supervised Embodied Active\nLearning (SEAL). It utilizes perception models trained on internet images to\nlearn an active exploration policy. The observations gathered by this\nexploration policy are labelled using 3D consistency and used to improve the\nperception model. We build and utilize 3D semantic maps to learn both action\nand perception in a completely self-supervised manner. The semantic map is used\nto compute an intrinsic motivation reward for training the exploration policy\nand for labelling the agent observations using spatio-temporal 3D consistency\nand label propagation. We demonstrate that the SEAL framework can be used to\nclose the action-perception loop: it improves object detection and instance\nsegmentation performance of a pretrained perception model by just moving around\nin training environments and the improved perception model can be used to\nimprove Object Goal Navigation.\n