Reinforcement Learning-based Decentralized Optimal Control for Large-Scale Multi-agent System by Using Neural Networks and Discrete-time Mean Field Games

Abstract

This paper proposed the Actor-critic-mass (ACM) algorithm to solve the discrete-time mean field type tracking control problem without discretization errors. The traditional large-scale multi-agent control problem suffers from computational explosion and communication difficulty due to the team's nearly infinite agent number. To deal with those challenges, the Mean Field Games (MFG) have been introduced in the multi-agent control problem to disconnect the computation and communication complexity with the agent number. However, traditional mean field type control problems mainly studied the continuous- times systems limited by the concerns of losing the solutions' existence or uniqueness caused by discretization. In this paper, the Feynman-Kac formula is utilized to reformulate the continuous MFG equation system into a backward discretized system. Specifically, the Hamiltonian-Jacobi-Bellman (HJB) equation and backward Kolmogorov equation in the form of a backward stochastic differential equation (BSDE) are introduced. Meanwhile, three neural networks (NN), i.e., the actor, critic, and mass NN, are developed to approximate the MFG equation system's discretized solution and the optimal control. Moreover, the stability and convergence of the NNs and the closed-loop system are provided via the Lyapunov stability analysis. Finally, a series of numerical simulations have shown the new discrete algorithm's performance.