Constrained Reinforcement Learning for Stochastic Dynamic Optimal Power Flow Control

Deep Reinforcement Learning (DRL) algorithms have become popular for solving power system control problems. In conventional DRL, the agent is incentivized to explore all policies that can be encoded in a neural network (NN) with the sole objective of maximizing the reward function. In doing so existing DRL algorithms can produce infeasible solutions, i.e. recommend actions that do not satisfy the power flow equations, voltage limits as well as dynamic constraints. Power systems, are safety critical systems and ensuring the aforementioned physical constraints are met is essential. Often the problem remedied by projecting the actions in the feasible set, which is suboptimum. In this paper, we propose a primal-dual approach to learn optimal constrained DRL policies for dynamic optimal power flow problems, aiming at controlling power generations and battery outputs. Case studies on the IEEE standard systems validate the superior performance of dynamically adapting the policy to the environment and the required safety levels.