Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programming

Safety assurance is uncompromisable in safety-critical environments, especially when drastic model uncertainties (e.g., distributional shift) exist, especially with humans in the loop. However, incorporating uncertainty in safe learning will naturally lead to a bi-level problem, where at the lower level, the worst-case safety constraint is evaluated within the uncertainty ambiguity set. In this paper, we present a tractable distributionally safe reinforcement learning framework that enforces safety under a distributional shift, as measured by a Wasserstein metric. To improve the tractability, we first use duality theory to transform the lower-level optimization from the infinite-dimensional probability space where distributional shift is measured, to a finite-dimensional parametric space. Moreover, by differentiable convex programming, the bi-level safe learning problem is further reduced to a single-level one with two sequential computationally efficient modules: a convex quadratic program to guarantee safety, followed by a projected gradient ascent to find the worst-case uncertainty simultaneously. This end-to-end differentiable framework with safety constraints offers a tractable single-level approach to addressing distributional safety. We test our approach on first- and second-order systems with varying complexities, including hardware demonstration on a 6-DOF drone. Compared with both uncertainty-agnostic policies and robust policies, our approach demonstrates a significant improvement in safety guarantees.