Multi-Agent Reinforcement Learning (MARL) methods have shown promise in enabling agents to learn a shared communication protocol from scratch and accomplish challenging team tasks. However, the learned language is usually not interpretable to humans or other agents not co-trained together, limiting its applicability in ad-hoc teamwork scenarios. In this work, we propose a novel computational pipeline that aligns the communication space between MARL agents with an embedding space of human natural language by grounding agent communications on synthetic data generated by embodied Large Language Models (LLMs) in interactive teamwork scenarios. Our results demonstrate that introducing language grounding not only maintains task performance but also accelerates the emergence of communication. Furthermore, the learned communication protocols exhibit zero-shot generalization capabilities in ad-hoc teamwork scenarios with unseen teammates and novel task states. This work presents a significant step toward enabling effective communication and collaboration between artificial agents and humans in real-world teamwork settings.
Paper
Similar papers
Peer review
Summary
The paper introduces a framework for communication in multi-agent RL. The communication space is aligned with the embedding space of natural language. The loss for training the underlying RL algorithm is augmented with a cosine similarity loss to align the communication message generated to the embeddings generated from LLMs.
Strengths
- The approach is quite flexible in the sense that it can be used with any standard RL algorithm - The loss function is quite simple but effective. - The proposed method shows better performance in task-related metrics and cosine similarity/BLEU scores compared to other baselines.
Weaknesses
- Figure 1 could have more details about the architecture in the caption. There is a reference to the figure on line 135 where it states that more information about the architecture can be found in Fig 1. But the caption and the figure were not clear and I had to read more about it in the text. - The experiments related to the zero-shot generalizability of the model are not convincing enough. I believe there isn’t much of a difference between $pp_{v0}$ and $pp_{v1}$ other than the vision range of the agents. Also, I am not sure if Table 3 just shows 4 instances of coming across different observations/locations in the environment. I am curious to see how this would perform in 10x10 test environments when they are trained on 5x5 grid environments. Will the communication messages be able to handle these new coordinates like (9,9), (7,8) which were previously unseen in the training scenarios? - I was not able to understand the exact setting used for the ad-hoc teamwork experiment. In what manner were the LLM agents controlled? How was the communication message decoded from LangGround to natural language? Was there a reason for evaluating the ad-hoc teamwork on just 24 episodes in Table 4? - The environments in which the model was tested are simple grid-world environments. The usage of LLMs to generate embedding data motivates testing it out in more complex environments (I think this has been pointed out in the future work section.).
Questions
Refer to the weaknesses section for my questions.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The experimental evaluation in some of the experiments Tables 3 and 4 (refer weaknesses section above) are not extensive. Also, the procedure to obtain the results in Table 4 is not clear.
Summary
This paper proposes a pipeline for aligning multi-agent communication with natural language by using an LLM to collect a synthetic dataset of natural language interactions. Within a MARL communication pipeline, this dataset is used to align the learned communication space with word embeddings of reference communications. This pipeline enables human interpretable messages and ad-hoc collaboration across multiple communication-intensive MARL environments.
Strengths
- The research direction proposed in this work is important since it connects the literature of emergent communication with natural language. - The paper is well written and easy to follow. - The proposed method is easily generalizable to new settings and is usable with different communication backbones and LLMs.
Weaknesses
- This work is missing a discussion comparing the proposed technique to the prior work of CICERO[1] from 2022. CICERO used a dataset of human messages and actions in real games to build a task-specific language model and trained a policy network that communicates by sharing embeddings that are translated through the LLM. This work has many components in common with CICERO and needs to discuss its novelty (beyond replacing human data with GPT-4 data). - The supervised loss seems to imply that the communication protocol is just a distillation of GPT-4. Although the RL objective could theoretically help determine what information is useful to share, it seems more likely that RL would just learn to optimize actions while leaving the communication protocol to the supervised loss. Even if the RL objective does influence the communication protocol, the only alternative messages it could generate are messages corresponding to different (observation, action) pairs, so I doubt the claim in line 175 that RL determines useful information to share with other agents. - That being said, I would agree with the notion that RL learns to "interpret" messages from other agents. - Section 6.3.1 seems misleading to me. Specifically, the way that communication vectors are translated to English sentences are via cosine similarity matching to the most similar message the dataset (according to lines 653-654 in the appendix). Therefore, the claim that communication messages are similar to natural language sentences in unseen locations (lines 285 to 287) must be true by construction. [1] FAIR et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074, 2022. doi: 10.1126/science.ade9097.
Questions
- The BLEU score in line 267 doesn't make sense to me. It is comparing the score between the natural language messages and reference messages in D with the most similar word embedding, but in the appendix (lines 653-654) it states the communication vectors are translated to English sentences via cosine similarity matching. In that sense, shouldn't the BLEU scores be perfect? - How does GPT-4 compare against your technique (and baseline emergent MARL communication) in the studied settings?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
The limitations mentioned in the discussion are solid. I would additionally include how this technique depends on the notion that the grounding LLM "knows" what message is optimal.
Author response pt. 1
We are sorry to see you have changed your overall rating from '5 = borderline accept' to '3 = reject' after reading our rebuttal and additional experiment results. We seek to provide additional clarifications for your remaining concerns. ## GPT-4 comparison We acknowledge your evaluation of the results in Table 4 that the LLM-only team performs on par with MARL-only teams. However, as pointed out in Section 6.3.2, the comparison among homogeneous teams is not the focus of ad-hoc teamwork experiments. Instead, we try to quantify the performance loss when introducing an unseen teammate into the team, and the degree of human-agent communication achieved via LangGround compared to random baselines, by comparing different ad-hoc teams (e.g., LLM+ aeComm, LLM+LangGround). Therefore, whether pure LangGround teams perform better than pure GPT-4 teams does not impact the conclusion we draw in the paper. We do not see this as a weakness of our work by any means. ## Communication protocol is a distillation of example messages We acknowledge your hypothesis about "distillation," as it is a possible analogy of our proposed pipeline. However, as mentioned in the overall rebuttal, it is impossible to fully decouple the contribution of RL and SL with our current model structure. We have no way to verify the hypothesis about "interpretation" and can only attribute both communication and action to RL and SL as a multi-objective optimization problem. More importantly, whether the hypothesis is true or false does not diminish the value of our work. Many previous works in multi-agent communication learn a latent representation of observation and directly use the encoding as a communication vector [1, 2]. Our method works similarly, but we use word embeddings of observation-action descriptions as the latent representation. Whether RL determines information to share or "interprets" messages are merely different technical approaches, rather than critical concerns that limit our pipeline. ## Communication sparsity We seek to clarify the misunderstanding you might have regarding the experiment settings and the value of sparse communication. 1. We used the semi-cooperative setting for predator-prey (i.e., mixed mode) where agents receive individual rewards. The original statement in IC3Net you cited only applies to fully-cooperative settings where agents receive a global shared reward. In fact, the communication probability learned by LangGround agents is around 80%, which aligns with the results reported in the IC3Net paper. 2. The value of sparse communication is well acknowledged by the research community. For example, IC3Net outperforms CommNet in mixed predator-prey by introducing the gating function to allow agents to learn to communicate when necessary. Previous research proves that a sparse communication protocol can be learned with little to no performance loss [3]. The statement that 'learned communications are often uninformative' is exactly the motivation for exploring sparse communication. We see this as the nature of agent and human communication, rather than a weakness of our work. We leave further exploration to future work because communication sparsity is out of the scope of this paper. ## LangGround relies on LLMs We acknowledge your statement that LangGround's communication depends on the quality of the example dataset. However, the collection of this dataset is fairly accessible. LLMs are known to have good linguistic capabilities (e.g., describing) while struggling with formal reasoning (e.g., planning). Our pipeline actually allocates the appropriate task to LLMs, i.e., describing the obs-action pairs, and leaving the planning part for RL. In more complicated scenarios in which communication is beyond describing observation and action, we could still expect LLMs to generate reasonable outputs since they were trained on massive conversations and dialogue data. Compared to alternative methods in embodied agents where LLMs must make correct action planning at every timestamp, it is more feasible to collecte semantically meaningful messages from either LLMs or any other sources. To be continued. [1] Learning to ground multi-agent communication with autoencoders. NeurIPS 2022 [2] Learning to Communicate using Contrastive Learning. ICLR 2024 [3] Towards True Lossless Sparse Communication in Multi-Agent Systems. ICRA 2023
## Compare LangGround with LLM embodied agents Although comparing LLM embodied agents with MARL agents (e.g., LangGround) is not the main focus of this paper, we provide our analysis in three parts: - Many studies evaluate LLM's planning capability for embodied agents. It is widely acknowledged that embodied LLM agents may generate infeasible action plans in interactive tasks due to hallucinations and bias from training data [1]. Consequently, MARL agents are generally more efficient in completing tasks than pre-trained LLM agents [2]. Specifically in our case, LLM and LangGround perform similarly because 1) LangGround is not 100% optimized for task reward, 2) LangGround does not use SOTA MARL models as the backbone (e.g., MAPPO), and 3) LLM embodied agents are heavily prompt-engineered for specific environments. - LLMs have a large number of parameters and thus take longer to run. In our case, calling OpenAI's API is ~50 times slower than running local inference of MARL models and costs ~5 dollars for each episode. Taking the massive time and resources required to train LLMs into consideration, MARL models are more affordable and eco-friendly in solving specific tasks. - The effort required for generalizing LangGround to different scenarios is smaller compared to embodied LLM methods. Because LangGround only uses effective communication messages from LLMs, a mere amount of prompt engineering is required compared to alternative methods that require LLMs to generate actionable plans for each step. ## Gating experiment We ran this ablation study to answer your original question about attributing RL to communication. We came to similar conclusions that MARL indeed learns to selectively communicate with the help of language grounding, i.e., mediator effect. We also find the results interesting and plan to further explore them in future work. It would be helpful for us to improve the work if you could specify the potential research directions you have in mind. ## Generalization We would like to clarify the experimental settings to help you better evaluate the zero-shot generalization results. The dataset $\mathcal{D}$ we used in the evaluation is constructed from limited trajectories of LLM agents, thus usually does not cover all (obs, action) combinations. For example, the full dataset of $pp_{v0}$ contains 1893 data points which covers only 45 out of 50 possible observations. The additional zero-shot generalization experiments were conducted on a 10 by 10 map with an even smaller $\mathcal{D}$ of size ~1000 data points, due to time and resource limitation. This means the 100% condition in the attached PDF represents roughly 12.5% (the map size is 4 times bigger, and the dataset is half the size) of all possible (o, a) combinations. With this being said, the generalization performance we observe in Table A is approximately between 12.5% to 3% states grounded. We will revise the description to make the experimental setting clearer. We hope this could help you calibrate your evaluation of the scalability of LangGround in more complex scenarios. [1] LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks. arXiv 2024 [2] Theory of Mind for Multi-Agent Collaboration via Large Language Models. EMNLP 2023
Thank you for the clarification regarding the latest generalization results! I see that this result is much stronger than I initially realized, demonstrating that the communication protocol has genuine non-trivial interpolating capabilities. I see this as a valuable result and will increase my score accordingly. Regarding the gating experiments for future work, I would've liked to see if the "communication" network could've been replaced entirely by the lookup procedure (or an approximate lookup if not exactly available) since this is treated as the ground truth in supervised learning. Furthermore, I would've liked qualitative observations regarding the discarded messages since these seem to have a large impact on task performance if left in.
Thanks for your timely reply. We appreciate your acknowledgment of the newly added experimental results and the suggestions for future directions. We will revise the paper accordingly.
Summary
The paper presents a novel computational pipeline aimed at aligning the communication space of Multi-Agent Reinforcement Learning (MARL) agents with an embedding space of human natural language. The authors propose grounding agent communications on synthetic data generated by embodied Large Language Models (LLMs) in interactive teamwork scenarios.
Strengths
- The use of synthetic data generated by LLMs to align MARL agents' communication is a creative application of existing technologies in a new way, demonstrating originality in methodology. - While the paper does not present new theoretical results, it does provide a solid empirical foundation for its approach, which is well-supported by experiments. - The proposed computational pipeline appears technically sound, with a clear explanation of how it aligns with human language and the rationale behind the design choices. - The authors provide clear explanations of complex concepts, such as the alignment of communication spaces and the grounding process, making the paper accessible to a broader audience.
Weaknesses
- The paper does not provide a theoretical framework or proofs to support the empirical findings. Developing a theoretical basis could strengthen the claims and provide deeper insights into why the approach works. - The experiments are conducted in controlled environments. To strengthen the claims, testing the approach in more diverse and complex scenarios could provide evidence of broader applicability. - The paper relies heavily on synthetic data generated by LLMs. There might be concerns about the representativeness of this data for real-world scenarios. - Some details regarding the implementation of the MARL agents and the interaction with LLMs could be better elaborated.
Questions
See weakness
Rating
6
Confidence
4
Soundness
3
Presentation
2
Contribution
2
Limitations
See weakness
Summary
The paper presents a method of using LLMs with synthetic data to generate human interpretable multi-agent communication protocols using zero-shot learning. The model, called language-grounded multi-agent communication (LangGround) aligns the communication of the multi-agent model with the LLM based communication. The model is tested on 2 versions of the Predator-Prey and 1 version of the USAR environments against multiple baselines, showing that the model presented in the paper performs comparable or better than state-of-the-art models in Multi-Agent Communication. Moreover, the paper shows that the communication protocol is generalizable and allows for Human interpretation and the ability to work in teams with LLM-based agents.
Strengths
The model is novel and experiments show that it outperforms other state of the art baselines. The related works is of sufficient length for a paper of given length. The approach is clear and the figures are easy to understand. Testing against multiple MARL Environments and configurations provide a convincing argument. The shared supplementary materials allow for easy reproducibility.
Weaknesses
There could be a Human Subjects experiment to show that the model that is presented is actually preferred by humans over alternative models. The experiments could be expanded to show the performance of the model at different scales. The differences between the model presented in the paper and [24] can be highlighted more. While section 4.1 goes into detail that the Language model used is derived from the previous work, it may be better to stress the primary differences given that both models are tested on the USAR environment.
Questions
How is this model more advantageous than a pure LLM based environment as presented in [24]? How open is this model to hallucinations from the LLM model? How does the performance of the LLM Model used affect the performance of the trained LangGround Model?
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Prompt engineering is required to allow for the model to be integrated into new environments. The model that is trained depends on the performance of the Language Model. The models are shown to trade off utility and informativeness, causing a decrease in performance for interpretability.
Thank you for your detailed rebuttal. I am satisfied with the answer to the question of Cosine similarity + BLEU scores, but I still have multiple lingering concerns. ## GPT-4 Comparison Thank you for pointing me to table 4; during my original reading of the paper, I thought the LLM row was another form of ad-hoc teamwork based on the title. However, I disagree with the analysis that LLM-only teams perform worse than MARL-comm teams given that it is significantly better in USAR and within the margin of error for the other two settings. ## Does RL Determine Useful Information From the overall rebuttal, it still seems like the overall communication protocol is just a distillation of OpenAI's text-embedding-3-large over a dataset of GPT-4 communication demonstrations, but RL helps interpret messages and determine whether to send messages in the first place. I am surprised by the fact that removing the gating function causes a significant drop in performance, since only cooperative settings are studied in this environment, so I don't know why always communicating would harm performance (in fact the original IC3Net paper states that setting communication probability to 1.0 is often beneficial in purely cooperative settings). This seems to imply that the communications are often distracting and uninformative. This result would need to be analyzed much more in the paper to explain the role of RL in communication. ## LangGround relies on LLM performance Based on the rebuttal, it seems fair to say that LangGround's *ad-hoc teamwork* abilities are impacted by the performance of the LLM (or the dataset generator more generally). If the LLM cannot effectively explain an observation-action pair, LangGround would inherit this weakness. Furthermore, if the LLM never experiences an observation-action pair, the lookup will force LangGround to copy the message from another observation-action pair in the dataset, limiting generalization in more complex settings where the LLM cannot cover the whole observation space. ## CICERO I generally agree with your summary of CICERO. However, I would still expect a comparison between these two techniques in the related works section. Both have separate modules for actions and conducting actual language. In CICERO, there is a dialogue-free RL model that is grounded to language via KL regularization against a dialogue-conditioned action model, which is comparable at a high level to your entire model. The language generation component is an LLM in CICERO that is conditioned on intents, whereas it is a database lookup in yours (which is not a key contribution based on the fact it is only explained in the end of the appendix). In my opinion, the key difference between CICERO and your work is the fact that the latent communication vector is an additional input to the RL agent instead of serving as an anchor policy. Regardless, CICERO is a key piece of related work that needs to be addressed in the related works section, especially given the last sentence of your introduction.
Author response pt. 2
## Generalize to unseen states We would like to first emphasize the main contribution of our paper: we propose a general pipeline to align agent communication space with the semantic space of human language by grounding them on a small set of instances. With a well-aligned communication space, any method can be used to translate the high-dimensional vectors into natural language messages. The cosine similarity matching method used in the paper is only for the purpose of proof-of-concept. Alternatively, we could learn a sequential model for this translation process for better generalizability. In the zero-shot experiments, we have demonstrated that LangGround performs reasonably well even when LLMs do not cover all states. We have also experimented with using a similarity measurement to approximately retrieve missing (action, obs) pairs from the dataset. Its performance is close to the current exact retrieval method while potentially generalizing better in complicated scenarios. Experimental results can be provided upon request. ## CICERO We acknowledge the high-level similarity between CICERO and LangGround, and will add the comparison to the related work section and revise the claim in the introduction. However, we would like to clarify the framing of our work to better differentiate CICERO from our approach. LangGround is an extremely flexible pipeline with most of the modules being interchangeable, such as the word embedding model, base MARL-comm model, message dataset source, and most importantly, the translation module. As discussed previously, we focus on communication alignment and empirically evaluate it in the context of ad-hoc teamwork. The cosine similarity matching method used for translation during evaluation is only for the purpose of proof-of-concept and is subject to change. With the above argument in mind, we believe the key difference between CICERO and LangGround is in the intermediate representations (i.e., intents in CICERO and comm vectors in LangGround) between the RL policy and translation module. In CICERO, the format of intents is defined by domain experts. An intent model is trained to identify players' intents from conversation history based on human-annotated data. In LangGround, we replace those procedures by using LLMs as the semantic knowledge base. This provides LangGround more flexibility to be applied to different domains and task scenarios. In addition, high-dimensional communication vectors are more descriptive than intents with fixed format, thus could potentially carry more contextual information from RL policy to language models. The topological similarity property of the latent communication space also allows LangGround to generalize to unseen states in a zero-shot manner. To summarize, we acknowledge the contribution of CICERO in the area of human-interpretable communication. However, we also believe that our pipeline, LangGround, is significantly different from CICERO and provides novel contributions to the community. ## Final comments We appreciate your time and effort in reading our rebuttal and providing feedback. We hope our clarifications address your concerns and kindly request that you reconsider the overall rating.
Thank you for the additional clarifications. I agree with the new response regarding CICERO and I appreciate the clarifications regarding communication sparsity. I will summarize my current thoughts on the paper below. 1. I consider prompting the LLM to be a baseline method given that it is used to generate the language dataset to begin with. Based on the results of this paper, it seems like using the LLM directly would still be better for solving these tasks. When I gave my original score, I thought there would be an argument that RL could enable significantly stronger actions than the baseline LLM (despite distilling the key communication protocol), but after reviewing Table 4 I saw that this wasn't the case in the studied environments. 2. The gating experiments in the rebuttal are interesting but underexplored. LangGround's communication probability is ~80%, but removing the gate entirely causes an extremely large reduction in performance whereas IC3Net seems to be barely impacted. This result is very surprising and could lead to evidence that simply replacing the communication protocol with the LLM/dataset would not perform as well as LangGround. 3. My key takeaway from this paper is that existing general-purpose LLMs and embedding models can be used to create a communication protocol on-par with SOTA tabula-rasa emergent communication techniques in the studied environments. The generalization results imply that interpolation does occur, though there is a significant drop in performance when only 1/4 of the states are grounded (and similarity metrics are unimpressive when 1/2 of the states or less are grounded), raising doubts about the scalability of this technique for more complex settings.
I appreciate the effort taken by the authors to clarify some of the questions I had. I am happy with the clarifications given and would like to increase my score from 6 to 7.
Thank you for your appreciation. We appreciate your time and efforts in reviewing our paper. We will incorporate your suggestions into the final paper.
Reviewer hS12
The authors have provided a detailed rebuttal. Can Reviewer hS12 share their thoughts?
Reviewer NCiL
The authors have provided a detailed rebuttal. Can Reviewer hS12 share their thoughts?
After reading the rebuttal of the authors and the discussion between the authors and other reviewers, I think my concerns are mainly addressed. I would raise my rating.
We appreciate again your time and effort for reviewing our paper. We are glad that you are satisfied with the rebuttal and appreciate your re-evaluation of our paper based on the clarifications provided during the discussion period.
Decision
Accept (poster)