Self-Driven Particles (SDP) describe a category of multi-agent systems common\nin everyday life, such as flocking birds and traffic flows. In a SDP system,\neach agent pursues its own goal and constantly changes its cooperative or\ncompetitive behaviors with its nearby agents. Manually designing the\ncontrollers for such SDP system is time-consuming, while the resulting emergent\nbehaviors are often not realistic nor generalizable. Thus the realistic\nsimulation of SDP systems remains challenging. Reinforcement learning provides\nan appealing alternative for automating the development of the controller for\nSDP. However, previous multi-agent reinforcement learning (MARL) methods define\nthe agents to be teammates or enemies before hand, which fail to capture the\nessence of SDP where the role of each agent varies to be cooperative or\ncompetitive even within one episode. To simulate SDP with MARL, a key challenge\nis to coordinate agents' behaviors while still maximizing individual\nobjectives. Taking traffic simulation as the testing bed, in this work we\ndevelop a novel MARL method called Coordinated Policy Optimization (CoPO),\nwhich incorporates social psychology principle to learn neural controller for\nSDP. Experiments show that the proposed method can achieve superior performance\ncompared to MARL baselines in various metrics. Noticeably the trained vehicles\nexhibit complex and diverse social behaviors that improve performance and\nsafety of the population as a whole. Demo video and source code are available\nat: https://decisionforce.github.io/CoPO/\n