Reducing Learning Difficulties: One-Step Two-Critic Deep Reinforcement Learning for Inverter-based Volt-Var Control

—A one-step two-critic deep reinforcement learning (OSTC-DRL) approach for inverter-based volt-var control (IB-VVC) in active distribution networks is proposed in this paper. First, the problem of IB-VVC is formulated as a one-step Markov decision process, which reduces the difficulties of the DRL learning task. Correspondingly, a one-step actor-critic DRL scheme is designed, which has a simpler structure and avoids the problem of Q-value over-estimation. Second, considering two objectives of VVC: minimizing power loss and eliminating voltage violations, we utilize two critics to approximate the rewards of two objectives separately, which reduces the difficulties of the approximation tasks of each critic. OSTC-DRL under the simper structure improves the approximation accuracy of critics, accelerates the convergence process, and improves the control performance. The OSTC-DRL approach cooperates well with any actor-critic DRL algorithms for the centralized IB-VVC problem, and two centralized DRL algorithms were taken as examples. The multi-agent OSTC-DRL approach is also developed and applied to the decentralized IB-VVC problem. Extensive simulation experiments show that the two proposed OSTC-DRL algorithms require fewer iteration times and return better results than the recent DRL algorithms, and the multi-agent OSTC-DRL algorithms work well for decentralized IB-VVC problems.