Feudal Multi-Agent Reinforcement Learning with Adaptive Network Partition for Traffic Signal Control

Multi-agent reinforcement learning (MARL) has been ap- plied and shown great potential in multi-intersections traffic signal control, where multiple agents, one for each intersec- tion, must cooperate together to optimize traffic flow. To encourage global cooperation, previous work partitions the traf- fic network into several regions and learns policies for agents in a feudal structure. However, static network partition fails to adapt to dynamic traffic flow, which will changes frequently over time. To address this, we propose a novel feudal MARL approach with adaptive network partition. Specifically, we first partition the network into several regions according to the traffic flow. To do this, we propose two approaches: one is directly to use graph neural network (GNN) to generate the network partition, and the other is to use Monte-Carlo tree search (MCTS) to find the best partition with criteria com- puted by GNN. Then, we design a variant of Qmix using GNN to handle various dimensions of input, given by the dy- namic network partition. Finally, we use a feudal hierarchy to manage agents in each partition and promote global coopera- tion. By doing so, agents are able to adapt to the traffic flow as required in practice. We empirically evaluate our method both in a synthetic traffic grid and real-world traffic networks of three cities, widely used in the literature. Our experimen- tal results confirm that our method can achieve better performance, in terms of average travel time and queue length, than several leading methods for traffic signal control.