Information Design in Multi-Agent Reinforcement Learning

Summary

This paper studied a Markov signaling game and proposed to use the signalling gradient and extended obedience constraint to build a learning system that can effectively train the sender in the game to reach desirable equilibriums. Experiments performed on two toy problems suggest that the newly developed SGOC algorithm can perform well and reach desirable equilibriums, compared to several competing learning methods.

Strengths

It is interesting and important to study the information sharing problem in a multi-agent context. The Markov signaling game presented in this paper appears to be of some practical values and deserves further investigation. The design of the new SGOC algorithm also introduces some novel ideas.

Weaknesses

It is not clear whether the Markov signaling game introduced in this paper is new and has never been studied previously. The novelty of this game may need to be clarified. Its usefulness for practical applications also seems uncertain. I cannot immediately relate the two toy problems studied experimentally in this paper with any practically important scenarios where an effective machine learning-based solution of the Markov signaling game is essential. Hence, the motivation and research value of this paper may need to be significantly strengthened. The above concern is partially triggered by the lack of discussion of related works in the main text of this paper. Because of this, the importance and novelty of studying the Markov signaling game remain doubtful to a certain extent. The true technical novelty of using the signaling gradient and the extended obedience constraint hence should be further investigated. It is assumed that the sender can access the receiver's policy and observation (the receiver's observation is not necessarily accessible to the sender). I have no idea when such an assumption could be true in a practical scenario. In fact, the calculation of the signaling gradient may rely on this assumption. However, when the sender and receiver have potentially conflicting interests, it is entirely unclear why the receiver would be motivated to honestly support the sender by providing its internal information to the sender (in a conflicting scenario, the sender and receiver may not have the motivation to participate in a centralized training process). This further makes me question the practical value of the new SGOC algorithm. Some statements in the paper sound a bit questionable and needs further clarification and justification. For example, the authors mentioned several times that the signaling gradient can alleviate the non-stationarity between the sender and receiver. However, it remains unknown to me why the non-stationarity can be alleviated in this way. Moreover, since the receiver is handling a partially-observable problem, it may consider to adopt a recurrent neural network as its policy network. In view of this, the calculation of the signaling gradient may become significantly more complicated. Meanwhile, the idea of biased policy gradient has been mentioned multiple times in this paper. However, I can hardly understand its true meaning. Hence, related discussions in the paper do not sound highly convincing to me. Besides the above, I have no idea why $\nabla_{\eta}\pi_{\theta}(a|o,\sigma)$ should be 0 and why the sender's signaling scheme and the receiver's action policy should be updated alternately rather than simultaneously. This as well as several other algorithmic design details appear to have some technical issues. The authors discussed the necessity of establishing a reputation system. However, the feasibility, complexity, and difficulty of building such a reputation system have not been investigated with sufficient depth. While building such a reputation system may not be necessary for this study, its feasibility could significantly affect the importance and practicality of the newly developed algorithm.

Questions

Why is the Markov signaling game studied in this paper new and practically important? What is the practical difficulty of supporting/realizing the commitment assumption? When the sender and receiver have potentially conflicting interests, why would the receiver be motivated to honestly support the sender by providing its internal information to the sender? Why will the non-stationarity issue be alleviated by using signalling gradient? What is the exact meaning of biased policy gradient? Why should $\nabla_{\eta}\pi_{\theta}(a|o,\sigma)$ be 0? Why should the sender's signaling scheme and the receiver's action policy be updated alternately rather than simultaneously? What are the feasibility, complexity, and difficulty of building a reputation system to support the basic assumptions of the new algorithm and the Markov signaling game?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I do not have any concerns regarding this question.

Summary

This paper proposes a new agent-interaction framework called Markov signaling games, and the corresponding algorithm for mixed-motive tasks. Various experiments and discussions are provided to show the efficacy of the proposed method.

Strengths

- The authors provide a new formulation of agent-interaction framework for multi-agent reinforcement learning and the corresponding detailed explanation. - The proposed algorithm to solve Markov signaling game with theoretical results is provided.

Weaknesses

It is unclear what makes Markov signaling games important. While related works on multi-agent RL with communication are provided, It seems that Markov signaling games appear to be just one concept of communication in multi-agent RL. In the proposed game, it is assumed that the sender observes a fully environmental state and sends a message to the receiver. Is this assumption reasonable and important? The authors should explain why Markov signaling games are important compared to conventional communication frameworks in multi-agent RL, where multiple agents aim to solve tasks while communicating with other agents. The conventional framework does not even differentiate between which agents are senders or receivers. Furthermore, the paper should include a comparison with state-of-the-art multi-agent communication algorithms such as TarMAC[1], ATOC[2], and others. Currently, the authors only provide a comparison with DIAL. [1] A. Das et al., "TarMAC: Targeted multi-agent communication" [2] J. Jiang et al., "Learning attentional communication for multi-agent cooperation"

Questions

See Weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations are provided in Appendix.

Summary

This paper addresses the problem of multi-agent reinforcement learning in environments with mixed motives, which lie on the spectrum from fully cooperative to fully competitive scenarios. Specifically, it studies the problem of information design in this setting: how a sender might craft messages so that the behaviour of a receiver is successfully influenced. There are non-trivial implications in this setup, including the non-stationarity introduced by the messages and the fact that the receiver may choose not to obey them. The authors propose an algorithm to derive a signaling policy that takes into account the effect of the messages on the receiver's policy, as well as a way to learn the receiver's policy in a way that provides compatibility between the incentives. The algorithm is validated on two types of environments with mixed incentives.

Strengths

Originality: the paper is original in its consideration of mixed-motive environments, which have not received much attention in MARL, and yet are highly relevant for practical scenarios. Quality: the paper is of good quality. It motivates the work well by drawing connections to computational economics. It proposes methods that are grounded and sound, and performs a reasonable evaluation. Clarity: the paper is well-structured and clear overall, although I was not able to understand some of the technical details as I am not familiar with the area in depth. Significance: the area is one of active interest in the NeurIPS community.

Weaknesses

W1. In my opinion, the primary weakness of this work is the strength of a particular assumption made by the authors: that the sender has full access to the receiver's policy and observation at every step. I am struggling to think of a scenario in which the assumption is suitable (since it amounts to global visibility) and yet the setting is not fully cooperative. What are some examples of scenarios in which this is the case? This aspect needs to be discussed more thoroughly. Otherwise, the proposed method may be of limited practical use.

Questions

QS1. Please see W1 above, I would appreciate an answer. QS2. In the evaluation, it is unclear to me what is the appropriate metric to judge the performance of the algorithm. Is it $r^i$, $r^j$, or $r^i + r^j$ (social welfare)? Can we even speak of a right metric given the incompatibility between the incentives? Given that e.g. DIAL reaches the best equilibrium for the receiver, are the methods even comparable, or is the evaluation merely meant to illustrate which equilibria are reached? The "goal" of the evaluation should be clarified. QS3. Note that the supplementary material also includes the main paper text, and that the main PDF is not rendered properly (text is inaccessible, references are not linked). Consider compiling the main paper and supplementary material separately instead of cutting the PDF. QS4. Some nitpicks: first phrase of the abstract, while RL definitely is inspired by reward-driven learning in humans and animals, it is not necessarily setting out to mimic it; line 90 nature -> environment?.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Code and detailed information is available, which means the work is reproducible. No limitations beyond those already discussed in previous points. The authors have discussed broader impacts. The supplementary material includes a high-quality discussion of possible extensions in great depth.

Summary

The article tackles the scenario where two agents, a sender $i$ and a receiver $j$, act in an environment. The sender $i$ can send a message to the receiver $j$ at each time step. The sender's goal is to learn what information to send to the receiver such that it influences the latter's policy to maximize its own return. The authors derive a learning signal analogous to the policy gradient in the single-agent setup, and they also propose a version with a supplementary obedience constraint.

Strengths

The strength of the paper lies in the extensive discussion of the setup and its challenges, and the different equilibria that could be reached by the two agents. These are also supported by empirical results.

Weaknesses

The scenario is limited to the case where the sender can backpropagate through the receiver's estimator, which looks far from a realistic setup. Formula (3) needs some introduction. It is not clear why that formula expresses an obedience constraint. Similarly, it would be helpful to restate the revelation principle, not just mention it.

Questions

What is incentive compatibility?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors mention some limitations, but I would add to that the need to backpropagate through the receiver's policy estimator.

Summary

This paper studies how multi-agent reinforcement learning agents influence each other by hsaring different kinds of information. Additionally, the authors study how different information affects the transition of the agents trajectories and what information can be ignored. To ensure efficient learning, a suitable gradient for the problem is also proposed.

Strengths

This paper is well written and reads well. It is also well structured and contains only few spelling mistakes. Some other strengths in particular: - Proposes markov signalling games - Provides an extensive theoretical analysis of the approach - Develop the notions of a signalling gradient

Weaknesses

1. some of the related works in the appendix could be moved (or summarised) to the main text; the provided related works in the main text are too scarce and do not provide enough relevant information that relates to the paper 2. the introduced markov signalling games seem to have some links with the famous Lewis games and other similar referencial games. Would be interesting to see a discussion around these concepts, as I cannot find it in the paper. Some related works also use reinforcement learning to learn communication involving similar games and concepts, such as [1], [2], [3], [4]. minor: - line 323: "ours" -> "our" Overall, this paper is interesting and well structured. Please find here and below some points that I would like the authors to comment on. [1] https://openreview.net/pdf?id=O5arhQvBdH [2] https://arxiv.org/pdf/1804.02341.pdf [3] https://arxiv.org/pdf/1804.03980.pdf [4] https://arxiv.org/pdf/1705.11192.pdf

Questions

1. in line 306 it is stated that the receivers action policy is implemented using A2C. Is this the case for the sender too? Or do they share the same network as in multiple MARL approaches? This should be clarified in the paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

1. It would be interesting to analyse the explainability of the messages exchanged by the receiver and speaker in these experiments, using the proposed method, as I can't find this is the paper and it is something very relevant when it comes to communication-driven games.

Thanks for the response

I thank the authors for their detailed response. As the authors have stated, I think that it is important to include these discussions and references around emergent communication and signalling games, since they are strongly related to what is being presented. After these discussions and assuming that this is now included, I have no further questions and will raise my score.

Post-rebuttal response to authors

Many thanks, I appreciate the very detailed and precise response. The Braess paradox example is a great one and, in my opinion, more intuitive than the recommendation letter example. Possibly adding it in the introduction can help clarify the setting. Re. QS2, I'd also suggest adding this detail to the manuscript. Given the current limited scalability of the method, I am sticking with the original score. However, the treatment of the problem is original and I would encourage the authors to pursue the information design problem in a larger-scale MARL setting.