Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

In reinforcement learning (RL), function approximation errors are known to easily lead to the <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-value overestimations, thus greatly reducing policy performance. This article presents a distributional soft actor–critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-value overestimations. We first discover in theory that learning a distribution function of state–action returns can effectively mitigate <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-value overestimations because it is capable of adaptively adjusting the update step size of the <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor–critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state–action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.