Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning

Value-based methods of multi-agent reinforcement learning (MARL), especially\nthe value decomposition methods, have been demonstrated on a range of\nchallenging cooperative tasks. However, current methods pay little attention to\nthe interaction between agents, which is essential to teamwork in games or real\nlife. This limits the efficiency of value-based MARL algorithms in the two\naspects: collaborative exploration and value function estimation. In this\npaper, we propose a novel cooperative MARL algorithm named as interactive\nactor-critic~(IAC), which models the interaction of agents from the\nperspectives of policy and value function. On the policy side, a multi-agent\njoint stochastic policy is introduced by adopting a collaborative exploration\nmodule, which is trained by maximizing the entropy-regularized expected return.\nOn the value side, we use the shared attention mechanism to estimate the value\nfunction of each agent, which takes the impact of the teammates into\nconsideration. At the implementation level, we extend the value decomposition\nmethods to continuous control tasks and evaluate IAC on benchmark tasks\nincluding classic control and multi-agent particle environments. Experimental\nresults indicate that our method outperforms the state-of-the-art approaches\nand achieves better performance in terms of cooperation.\n