Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values

While Explainable Artificial Intelligence (XAI) is increasingly expanding\nmore areas of application, little has been applied to make deep Reinforcement\nLearning (RL) more comprehensible. As RL becomes ubiquitous and used in\ncritical and general public applications, it is essential to develop methods\nthat make it better understood and more interpretable. This study proposes a\nnovel approach to explain cooperative strategies in multiagent RL using Shapley\nvalues, a game theory concept used in XAI that successfully explains the\nrationale behind decisions taken by Machine Learning algorithms. Through\ntesting common assumptions of this technique in two cooperation-centered\nsocially challenging multi-agent environments environments, this article argues\nthat Shapley values are a pertinent way to evaluate the contribution of players\nin a cooperative multi-agent RL context. To palliate the high overhead of this\nmethod, Shapley values are approximated using Monte Carlo sampling.\nExperimental results on Multiagent Particle and Sequential Social Dilemmas show\nthat Shapley values succeed at estimating the contribution of each agent. These\nresults could have implications that go beyond games in economics, (e.g., for\nnon-discriminatory decision making, ethical and responsible AI-derived\ndecisions or policy making under fairness constraints). They also expose how\nShapley values only give general explanations about a model and cannot explain\na single run, episode nor justify precise actions taken by agents. Future work\nshould focus on addressing these critical aspects.\n