Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments

Exploration under sparse reward is a long-standing challenge of model-free\nreinforcement learning. The state-of-the-art methods address this challenge by\nintroducing intrinsic rewards to encourage exploration in novel states or\nuncertain environment dynamics. Unfortunately, methods based on intrinsic\nrewards often fall short in procedurally-generated environments, where a\ndifferent environment is generated in each episode so that the agent is not\nlikely to visit the same state more than once. Motivated by how humans\ndistinguish good exploration behaviors by looking into the entire episode, we\nintroduce RAPID, a simple yet effective episode-level exploration method for\nprocedurally-generated environments. RAPID regards each episode as a whole and\ngives an episodic exploration score from both per-episode and long-term views.\nThose highly scored episodes are treated as good exploration behaviors and are\nstored in a small ranking buffer. The agent then imitates the episodes in the\nbuffer to reproduce the past good exploration behaviors. We demonstrate our\nmethod on several procedurally-generated MiniGrid environments, a\nfirst-person-view 3D Maze navigation task from MiniWorld, and several sparse\nMuJoCo tasks. The results show that RAPID significantly outperforms the\nstate-of-the-art intrinsic reward strategies in terms of sample efficiency and\nfinal performance. The code is available at https://github.com/daochenzha/rapid\n