Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Tasks

There is a growing interest in the community in making an embodied AI agent\nperform a complicated task while interacting with an environment following\nnatural language directives. Recent studies have tackled the problem using\nALFRED, a well-designed dataset for the task, but achieved only very low\naccuracy. This paper proposes a new method, which outperforms the previous\nmethods by a large margin. It is based on a combination of several new ideas.\nOne is a two-stage interpretation of the provided instructions. The method\nfirst selects and interprets an instruction without using visual information,\nyielding a tentative action sequence prediction. It then integrates the\nprediction with the visual information etc., yielding the final prediction of\nan action and an object. As the object's class to interact is identified in the\nfirst stage, it can accurately select the correct object from the input image.\nMoreover, our method considers multiple egocentric views of the environment and\nextracts essential information by applying hierarchical attention conditioned\non the current instruction. This contributes to the accurate prediction of\nactions for navigation. A preliminary version of the method won the ALFRED\nChallenge 2020. The current version achieves the unseen environment's success\nrate of 4.45% with a single view, which is further improved to 8.37% with\nmultiple views.\n