Learning Multi-Agent Communication with Contrastive Learning

Summary

This paper studies multi-agent communication by contrastive learning. It is motivated by the fact that communicated messages based on local observation can be viewed as incomplete views of the global state. From this perspective, a contrastive learning based approach is proposed, where states from close time steps are considered as positive samples, and those from distant time steps or episodes are considered as negative samples. The proposed algorithm is tested on several multi-agent benchmarks.

Strengths

- The paper has an innovative perspective on the multi-agent communications, which motivates the use of contrastive learning. - The paper is well-written, and the proposed contrastive learning framework is easy and clean to implement. - There are various ablation studies over different components of the proposed algorithm and visualizations over the learned communicated messages.

Weaknesses

- The improvement over the baselines is not obvious, especially in Traffic Junction in Figure 2. The standard error is also too large with a lot of overlaps, so it may need more seeds of experiments. - The proposed algorithm CACL is only tested on three tasks. The paper could benefit from additional experiments on some more challenging tasks with partial observability where communications are intuitively beneficial.

Questions

- The policy $\pi$ defined in section 3 seems to be only conditional on the local observation $\tau^i$. Should it also be conditional on the communicated messages? - If some local observations miss important information, the approximated global states reconstructed by the message encoders may be very different from the true ones. Will this make the contrastive learning not meaningful? - Does CACL require all-to-all communications, i.e., each agent communicate to all the other agents? If so, CACL is not scalable with large number of agents. - During a time step, each agent receives multiple approximations of the global state from the communicated messages. This seems to include redundant information if the messages are not selectively received. How does CACL handle redundant information?

Rating

5: marginally below the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

The authors describe CACL, a contrastive learning approach for inducing communication among multiple agents. There are close parallels to classic contrastive learning methods in vision, for example, but the authors apply their technique to "emergent communication" to allow teams of agents to communicate with each other. "Postive" examples are grouped based on a window of recent timesteps, and, as in standard constrastive learning, agents learn to encode positive examples near each other. In experiments, the authors show that CACL outperforms numerous baselines, including the SOTA AEComm method (which in some ways is similar in that it is a non-reward-based mechansim for inducing emergent communication).

Strengths

I like this paper. It presents a simple idea that works well. ## Originality Applying contastive losses to emergent communication is somewhat novel. (I know other works have also come out in this area, but they remain different in some important ways). ## Quality The work is well-scoped and presented, with good results backing up claims. ## Clarity I find the paper quite clear. Some figures could likely be redone to present the same information better (e.g., Figure 3), but mostly these are small changes. ## Significance I think this work, should it be published, would be an important baseline for future emergent communication work.

Weaknesses

Overall, this is a strong paper. To further improve the paper the authors could 1) Conduct further experiments to fill in Figure 4 in more detail (instead of just 3 or 4 checkpoints along the curve) 2) Run more trials, especially in the traffic junction where variance is high and not all methods seem to have converged.

Questions

I have no outstanding questions. Overall, this was a clear paper.

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Summary

This paper proposes a new method for fully independent communication in MARL, CACL. The proposed method leverages the power of contrastive learning to ground communication and learning efficient communication for MARL tasks.

Strengths

This paper tackles the problem of communication for fully independent learners, which is a very important topic in MARL and it is often underexplored. Also, mixing contrastive learning with MARL is interesting. Generally, the paper is well organised and well written.

Weaknesses

Overall, this paper is interesting and investigates an important topic in MARL. However, I still have some concerns and questions that I would like the authors to comment on. Please find my comments below and questions ahead. * The example of predator prey in figure 1 (right) is a bit confusing. I would not agree that the given examples correspond to similar views; for example, the first view (counting from the top) seems more similar to the third view rather than to the second view. * In section 1, the authors mention: "we propose that an agent’s observation is a “view” of the environment state. Thus, different agents’ messages are encodings of different incomplete “views” of the same underlying state.". This is in fact the premise of a dec-pomdp; Observations are tipically local perceptions of the environment's state; in other communication methods in MARL where observation encodings are used as messages, I would say that the same logic is followed: individual perceptions of the environment are being shared as message encodings to the others. As described by the authors is section 3, the observations come from a function of the state. * In the loss function, the RL loss is not defined. It would be good to have it for clarity purposes. * A more detailed diagram of the architecture of this method could be beneficial to get a better understanding of the approach; the one presented in figure 8 in the appendix seems very simple and lacks detail and we cannot clearly understand how the gradients flow; this can be important since the authors are dealing with fully independent learning, without sharing parameters. Minor: * In section 3, do the authors mean $m_{t-1}^{-i}$ instead of $m_{t-1}^{-1}$? * In section 3 "At each time step, each agent $i \in N$ chooses an action $a ∈ A^i$": shoud be $a^i$ since ahead $a$ is defined as the joint action.

Questions

* I have questions about whether it is reasonable to evaluate the similarity of messages of different agents by simply looking at a window of a few timesteps in the trajectory. The observations corresponding to the generated messages can be different in important aspects from one timestep to the other, and thus would require distinct messages that could be biased by the contrastive loss. I am unsure whether this would scale to more complex cases, since it could not capture these differences in the observations. * The experimented environments seem a bit simple and model scenarios where the observations can indeed be more similar to each other in some cases. Have the authors tested in more complex scenarios where there can be stronger variations on the observations such as SMAC? It would be interesting to see the performance in such complex environments. * The authors mention that the setting followed is a fully decentralised setting where the agents do not share parameters or gradients (section 1); does this apply to the message encoder? I.e., do the agents share the same message encoder or does each one of them use a separate encoder to generate messages? * I believe another potential direction for further work would be to investigate how to make methods such as the proposed one work together with the reward. I.e., in section 5.5 it was shown that currently it is detrimental. Yet, have the authors thought whether the method can be improved in any way in order to take advantage of the reward as well?

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Summary

This paper presents a novel approach to guide multi-agent communication learning via contrastive learning in a decentralized MARL scenario. The intuition is that under similar circumstances the agents should emit similar messages, and vice versa. Hence the authors employ contrastive learning to maximize the mutual information between messages of a given trajectory and minimize other cases. The authors claimed their method has outperformed exisiting approaches on several benchmarks.

Strengths

The idea delivered by this work is clear and somewhat grounded. Indeed it would be worthwhile for agent to learn a guidance of its message during multi-agent communication. And the intuition of enforcing messages under similiar state to be alike with each other is a straightforward motivation, for which contrastive learning might be one of the most popular method to achieve.

Weaknesses

However, after going through the whole paper, It is easy to find that the proposed idea is less sufficiently proved and there are many flaws in the manuscript. There are a few such perspectives: 1. In section 4, the negative samples are defined as from outside the current time window or other trajectories. This is not technically sound since it would be possible for agents to encounter similar states at different trajectories (which would be considered as negative by the proposal). It is suggested that the authors should discuss such cases in detail and figure out more solid principle to decide positive/negative samples for contrastive learning. 2. The selected benchmark for comparison is kind of limited. Public MARL evaluation platforms like SMAC[1] or MATE[2] which involves higher complexity should be considered for more pursuasive comparison. In addition, in the compared task of Traffic Junction, the improvement seems to be marginal. 3. The compared baseline methods are sort of obselete. Newer published works in recent 3 years should get included. (especially new work in 2022-2023). [1] Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., ... & Whiteson, S. (2019). The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043. [2] Pan, X., Liu, M., Zhong, F., Yang, Y., Zhu, S. C., & Wang, Y. (2022). Mate: Benchmarking multi-agent reinforcement learning in distributed target coverage control. Advances in Neural Information Processing Systems, 35, 27862-27879.

Questions

More comprehensive comparison and analysis are expected: 1. It is encouraged for the authors to demonstrate the scalability of the proposed approach, like in a continuous state/action space environments which may involve a large number of agents with quite dense communication load. In such cases, would the proposed scheme be better than the most recent communication-based work like ATOC[1], MF-MARL[2], TarMAC[3], I2C[4], ToM2C[5]? 2. Besides showing the similarity of messages among multiple states, the exact improvement from such a contrastive learning method should be analyzed. For instance, it is better to compare the adjacency/disparity of positive/negative sample pairs before/after the proposed training. [1] Jiang, J., & Lu, Z. (2018). Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31. [2] Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., & Wang, J. (2018, July). Mean field multi-agent reinforcement learning. In International conference on machine learning (pp. 5571-5580). PMLR. [3] Das, A., Gervet, T., Romoff, J., Batra, D., Parikh, D., Rabbat, M., & Pineau, J. (2019, May). Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning (pp. 1538-1546). PMLR. [4] Ding, Z., Huang, T., & Lu, Z. (2020). Learning individually inferred communication for multi-agent cooperation. Advances in Neural Information Processing Systems, 33, 22069-22079. [5] Wang, Y., Zhong, F., Xu, J., & Wang, Y. (2021). Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. arXiv preprint arXiv:2111.09189.

Rating

5: marginally below the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

2 fair

Updated Paper

We would like to thank the reviewers for their time and insightful feedback. We are glad they find our work “grounded” (sueF), “an important topic” (64q7), “novel” / “innovative” (akDr, Y1JN) and “good results backing up claims” (akDr). We have updated the manuscripts according to their suggestions, here is a summary of the main changes: - Updated Figure 1 to clarify our connection to image multi-view learning, pointed out by Reviewer 64q7 - Added RL loss equations to Appendix A.1., pointed out by Reviewer 64q7 - Updated architecture figure to indicate gradient flow in Figure 8, Appendix A.3, pointed out by Reviewer 64q7 - Added a new plot of the contrastive loss over training in Figure 9 in Appendix A.4, to show how positive/negative samples become more separable, as requested by Reviewer sueF