Reinforcement Learning vs. Backstepping Control of Stop-and-Go Traffic

This article develops a Reinforcement Learning (RL) boundary controller of stop-and-go traffic congestion on a freeway segment. The traffic dynamics are governed by a macroscopic Aw-Rascle-Zhang (ARZ) model, consisting of $2\times 2$ nonlinear Partial Differential Equations (PDEs) for traffic density and velocity. The boundary actuation of traffic flow is implemented with ramp-metering, which is a common approach to freeway congestion management. We use a discretized ARZ PDE model to describe the macroscopic freeway traffic environment for a stretch of freeway, and apply deep RL to develop continuous control on the outlet boundary. The control objective is to achieve $L^2$ norm regulation of the traffic state to a spatially uniform density and velocity. A recently developed neural network based policy gradient algorithm is employed, called proximal policy optimization. For comparison, we also consider an open-loop controller and a PDE backstepping approach. The backstepping controller is a model-based approach recently developed by the co-authors. Ultimately, we demonstrate that the RL approach nearly recovers the control performance of the model-based PDE backstepping approach, despite no \textit{a priori} knowledge of the traffic flow dynamics.