Improving Robustness of Deep Reinforcement Learning Agents: Environment Attack based on the Critic Network

To improve robustness of deep reinforcement learning agents, a line of recent works focus on producing disturbances of the dynamics of the environment. Existing approaches of the literature to generate such disturbances are environment adversarial reinforcement learning methods. These methods set the problem as a two-player game between the protagonist agent, which learns to perform a task in an environment, and the adversary agent, which learns to disturb the dynamics of the considered environment to make the protagonist agent fail. Alternatively, we propose to build on gradient-based adversarial attacks, usually used for classification tasks for instance, that we apply on the critic network of the protagonist to identify efficient disturbances of the dynamics of the environment. Rather than training an adversary agent, which usually reveals as very complex and unstable, we leverage the knowledge of the critic network of the protagonist, to dynamically increase the complexity of the task at each step of the learning process. We show that our method, while being faster and lighter, leads to significantly better improvements in robustness of the policy than existing methods of the literature.