TCRL: Temporal-Coupled Adversarial Training for Robust Constrained Reinforcement Learning in Worst-Case Scenarios

Constrained Reinforcement Learning (CRL) aims to optimize decision-making policies under constraint conditions, making it highly applicable to safety-critical domains such as autonomous driving, robotics, and power grid management. However, existing robust CRL approaches predominantly focus on single-step perturbations and temporal-independent adversarial models, lacking explicit modeling of temporal-coupled perturbations robustness. To tackle these challenges, we propose TCRL, a novel temporal-coupled adversarial training framework for robust constrained reinforcement learning (CRL) in worst-case scenarios. First, TCRL introduces a worst-case-perceived cost perturbation function that estimates safety costs under temporal-coupled perturbations towards the need for a dual-constraint model adversarial attacks. Second, TCRL establishes a dual-constraint defense mechanism towards the countered temporal-coupled adversaries while maintaining the unpredictability of the reward. The experimental results demonstrate that TCRL consistently outperforms existing methods in terms of robustness against temporal-coupled perturbation attacks across a variety of CRL tasks. A detailed version with full theoretical analysis, extended experiments, and additional implementation details is available at: https://github.com/biubiubiubiuhub/TCRL/tree/master