MICo: Improved representations via sampling-based state similarity for Markov decision processes

We present a new behavioural distance over the state space of a Markov\ndecision process, and demonstrate the use of this distance as an effective\nmeans of shaping the learnt representations of deep reinforcement learning\nagents. While existing notions of state similarity are typically difficult to\nlearn at scale due to high computational cost and lack of sample-based\nalgorithms, our newly-proposed distance addresses both of these issues. In\naddition to providing detailed theoretical analysis, we provide empirical\nevidence that learning this distance alongside the value function yields\nstructured and informative representations, including strong results on the\nArcade Learning Environment benchmark.\n