Munchausen Reinforcement Learning

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most\nalgorithms, based on temporal differences, replace the true value of a\ntransiting state by their current estimate of this value. Yet, another estimate\ncould be leveraged to bootstrap RL: the current policy. Our core contribution\nstands in a very simple idea: adding the scaled log-policy to the immediate\nreward. We show that slightly modifying Deep Q-Network (DQN) in that way\nprovides an agent that is competitive with distributional methods on Atari\ngames, without making use of distributional RL, n-step returns or prioritized\nreplay. To demonstrate the versatility of this idea, we also use it together\nwith an Implicit Quantile Network (IQN). The resulting agent outperforms\nRainbow on Atari, installing a new State of the Art with very little\nmodifications to the original algorithm. To add to this empirical study, we\nprovide strong theoretical insights on what happens under the hood -- implicit\nKullback-Leibler regularization and increase of the action-gap.\n