Optimistic Agent: Accurate Graph-Based Value Estimation for More Successful Visual Navigation

We humans can impeccably search for a target object, given its name only,\neven in an unseen environment. We argue that this ability is largely due to\nthree main reasons: the incorporation of prior knowledge (or experience), the\nadaptation of it to the new environment using the observed visual cues and most\nimportantly optimistically searching without giving up early. This is currently\nmissing in the state-of-the-art visual navigation methods based on\nReinforcement Learning (RL). In this paper, we propose to use externally\nlearned prior knowledge of the relative object locations and integrate it into\nour model by constructing a neural graph. In order to efficiently incorporate\nthe graph without increasing the state-space complexity, we propose our\nGraph-based Value Estimation (GVE) module. GVE provides a more accurate\nbaseline for estimating the Advantage function in actor-critic RL algorithm.\nThis results in reduced value estimation error and, consequently, convergence\nto a more optimal policy. Through empirical studies, we show that our agent,\ndubbed as the optimistic agent, has a more realistic estimate of the state\nvalue during a navigation episode which leads to a higher success rate. Our\nextensive ablation studies show the efficacy of our simple method which\nachieves the state-of-the-art results measured by the conventional visual\nnavigation metrics, e.g. Success Rate (SR) and Success weighted by Path Length\n(SPL), in AI2THOR environment.\n