Modelling and exploiting teammates' policies in cooperative multi-agent\nsystems have long been an interest and also a big challenge for the\nreinforcement learning (RL) community. The interest lies in the fact that if\nthe agent knows the teammates' policies, it can adjust its own policy\naccordingly to arrive at proper cooperations; while the challenge is that the\nagents' policies are changing continuously due to they are learning\nconcurrently, which imposes difficulty to model the dynamic policies of\nteammates accurately. In this paper, we present \\emph{ATTention Multi-Agent\nDeep Deterministic Policy Gradient} (ATT-MADDPG) to address this challenge.\nATT-MADDPG extends DDPG, a single-agent actor-critic RL method, with two\nspecial designs. First, in order to model the teammates' policies, the agent\nshould get access to the observations and actions of teammates. ATT-MADDPG\nadopts a centralized critic to collect such information. Second, to model the\nteammates' policies using the collected information in an effective way,\nATT-MADDPG enhances the centralized critic with an attention mechanism. This\nattention mechanism introduces a special structure to explicitly model the\ndynamic joint policy of teammates, making sure that the collected information\ncan be processed efficiently. We evaluate ATT-MADDPG on both benchmark tasks\nand the real-world packet routing tasks. Experimental results show that it not\nonly outperforms the state-of-the-art RL-based methods and rule-based methods\nby a large margin, but also achieves better performance in terms of scalability\nand robustness.\n