CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee

In safe reinforcement learning (SRL) problems, an agent explores the\nenvironment to maximize an expected total reward and meanwhile avoids violation\nof certain constraints on a number of expected total costs. In general, such\nSRL problems have nonconvex objective functions subject to multiple nonconvex\nconstraints, and hence are very challenging to solve, particularly to provide a\nglobally optimal policy. Many popular SRL algorithms adopt a primal-dual\nstructure which utilizes the updating of dual variables for satisfying the\nconstraints. In contrast, we propose a primal approach, called\nconstraint-rectified policy optimization (CRPO), which updates the policy\nalternatingly between objective improvement and constraint satisfaction. CRPO\nprovides a primal-type algorithmic framework to solve SRL problems, where each\npolicy update can take any variant of policy optimization step. To demonstrate\nthe theoretical performance of CRPO, we adopt natural policy gradient (NPG) for\neach policy update step and show that CRPO achieves an\n$\\mathcal{O}(1/\\sqrt{T})$ convergence rate to the global optimal policy in the\nconstrained policy set and an $\\mathcal{O}(1/\\sqrt{T})$ error bound on\nconstraint satisfaction. This is the first finite-time analysis of primal SRL\nalgorithms with global optimality guarantee. Our empirical results demonstrate\nthat CRPO can outperform the existing primal-dual baseline algorithms\nsignificantly.\n