Summary
This paper introduces SeqComm, a novel multi-level communication scheme for multi-agent coordination in reinforcement learning. The key contributions are:
1. A new approach treating agents asynchronously with two communication phases: negotiation and launching.
2. Theoretical analysis proving monotonic improvement and convergence of learned policies.
3. Empirical evaluation on StarCraft multi-agent challenge v2 (SMACv2) showing superior performance over existing methods.
The paper addresses the coordination problem in multi-agent settings by allowing agents to communicate actual actions and determine decision-making priorities dynamically.
Strengths
The paper provides a solid theoretical foundation with proofs for policy improvement and convergence (Propositions 1 and 2, Theorem 1). The experimental methodology is sound, using the challenging SMACv2 benchmark and comparing against relevant baselines. The ablation studies support the importance of the proposed components.
The paper is generally well-written and organized. The introduction clearly motivates the problem and contributions. The method section provides a detailed explanation of SeqComm. Figures and algorithms help illustrate the approach.
The paper presents a novel and significant contribution to multi-agent reinforcement learning. The idea of asynchronous decision-making with dynamic priority determination is original and addresses an important challenge in the field. The theoretical guarantees and strong empirical results on a challenging benchmark demonstrate the value of the approach.
Overall the strengths are:
- Novel multi-level communication scheme addressing the coordination problem
- Theoretical analysis providing performance guarantees
- Fairly strong empirical results on SMACv2, outperforming existing methods
- Ablation studies demonstrating the importance of key components
- Addresses both full and local communication scenarios
I'd also like to point out that this was run on a GTX 1050, which makes the model approachable to a general audience.
Weaknesses
- The reliance on homogeneous agents with parameter sharing may limit the applicability of SeqComm in real-world scenarios where agents often have diverse capabilities. However, it is a commonly accepted assumption.
- The approach heavily relies on a world model, but there's limited discussion on how the quality of this model affects performance, especially in more complex environments. Also the effect of the attention module is not analysed thoroughly. Attention mechanisms can be computationally expensive, especially as the number of agents increases. The paper doesn't address how this affects the overall scalability of SeqComm. It's not entirely clear how the attention mechanism is adapted for the local communication scenario, where agents only communicate with nearby agents. Furthermore, attention weights could potentially provide insights into which agents or information are most important for decision-making, but this aspect isn't explored in the paper.
- Overall, I believe a table reporting IQM values over all environments would strengthen the paper and help with reproducibility. Furthermore, plots showing wall-clock time in comparison to other models would be insightful, especially since final performance only appears to be significantly better in p_10v10 and t_10v10.
- While the paper addresses local communication, there's limited discussion on the trade-offs between communication frequency, accuracy, and overall system performance.
- Though not standard in related work either, the paper doesn't thoroughly explore the robustness of SeqComm to noisy observations, communication failures, or adversarial agents, which are common challenges in real-world multi-agent systems.
- While TarMAC is a strong baseline, the paper could benefit from comparisons with more recent state-of-the-art methods in multi-agent communication.
Questions
1. How does the computational complexity of SeqComm scale with the number of agents, particularly in the negotiation phase? Is there a point where the overhead becomes prohibitive?
2. Have you explored the performance of SeqComm in heterogeneous agent scenarios where parameter sharing may not be applicable?
3. How might the method be adapted for such settings?
4. How sensitive is the approach to the quality of the world model, especially in more complex environments? What happens if the world model is significantly inaccurate?
5. The paper mentions that SeqComm can provide a proper priority of decision-making. How does this prioritization mechanism perform in highly dynamic environments where optimal decision order may change rapidly?
6. Have you investigated the potential emergence of behavioral patterns or strategies across agents due to the asynchronous decision-making process? Are there any interesting emergent behaviors?
7. How does the attention mechanism perform compared to simpler aggregation methods? An ablation study could illuminate this.
8. Does the attention mechanism learn meaningful patterns of inter-agent importance? Visualizing attention weights could provide insights into learned coordination strategies.
9. How does the computational cost of the attention mechanism scale with the number of agents, especially in the full communication scenario?
10. Could more advanced attention mechanisms (e.g., multi-head attention) provide further improvements?
11. How is the attention mechanism modified to handle the local communication scenario effectively?
Limitations
- The evaluation is limited to one type of environment (SMACv2). It's unclear how well the method generalizes to other multi-agent scenarios with different dynamics or objectives.
- The heavy dependence on a world model could be a significant limitation in environments where accurate modeling is challenging or computationally expensive.
- While the paper addresses local communication, there's limited discussion on the trade-offs between communication frequency, accuracy, and overall system performance.
- The empirical results are not significantly better in a lot of environments and changing the sight range from SmacV2 makes comparisons to previous work harder.