Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

In multi-agent games, the complexity of the environment can grow\nexponentially as the number of agents increases, so it is particularly\nchallenging to learn good policies when the agent population is large. In this\npaper, we introduce Evolutionary Population Curriculum (EPC), a curriculum\nlearning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by\nprogressively increasing the population of training agents in a stage-wise\nmanner. Furthermore, EPC uses an evolutionary approach to fix an objective\nmisalignment issue throughout the curriculum: agents successfully trained in an\nearly stage with a small population are not necessarily the best candidates for\nadapting to later stages with scaled populations. Concretely, EPC maintains\nmultiple sets of agents in each stage, performs mix-and-match and fine-tuning\nover these sets and promotes the sets of agents with the best adaptability to\nthe next stage. We implement EPC on a popular MARL algorithm, MADDPG, and\nempirically show that our approach consistently outperforms baselines by a\nlarge margin as the number of agents grows exponentially.\n