Summary
This paper considers the problem of safe/constrained multi-agent reinforcement learning (MARL) with general utilities and constraints, which could be generic functions of agents' occupancy measures. A distributed method is proposed for solving the constrained MARL problem with local constraints, where each agent can communicate to its few-hop neighbors in the underlying network graph topology. The proposed algorithm is theoretically analyzed in terms of convergence behavior and sample complexity, and numerical results are provided to demonstrate the efficacy of the proposed method in multiple constrained MARL environments.
Strengths
- The problem of constrained/safe MARL is of great practical interest and importance, so the paper focuses on a timely topic of interest to the NeurIPS audience.
- Distributed training of safe RL policies in multi-agent settings is of significance, and the paper makes important contributions in this direction.
- Extensive theoretical and numerical analysis is provided to showcase the benefits of the proposed approach.
Weaknesses
- The presentation and structure of the paper could be improved, especially since many of the important parts of the paper (especially the algorithm pseudo-code, flowchart, and numerical results) are deferred to the appendix. Mindful of the space limitations, it would be great if the authors could shuffle the contents in their revision to include the pseudo-code/flowchart and more numerical results in the main body of the paper, with at least one result for each environment introduced in Section 5.
- The authors mention [25] (Lu et al., AAAI'21) in their discussion of related work, but I would have liked a more extensive discussion on the similarities and differences between this work and [25], especially since they both consider an underlying graph topology, as well as numerical comparisons with [25] in addition to MAPPO-Lagrangian.
- The paper mostly focuses on tabular MARL settings with a finite state space. It is unclear how the proposed method generalizes to continuous state spaces.
Questions
- I believe the Transition decomposition assumption in lines 106-108 is very strong. Could you please comment on how you justify this assumption in practical settings and whether it could be relaxed to a weaker assumption?
- How does the proposed method extend to the case where the objective in Eq. (5) is not separable?
- Intuitively, the upper bound in Eq. (13) should depend on $\omega$ and $\chi$ in Assumption 3.2, so please discuss why they do not appear here.
- In Eq. (15), could $1/n$ be replaced with $1/|\mathcal{N}_i^{\kappa}|$? I understand that this is simply a scaling factor, but truncating the policies to $\kappa$-hop neighbors could also justify the aforementioned replacement, where the Lagrangian is always *averaged* over the $\kappa$-hop neighborhood of each agent.
- Does Fig. 1-d show that the constraints are always violated? Does this mean that no feasible policy in this environment might exist?
- In the Appendix, it is shown that MAPPO-L performs worse than the proposed method. Could you provide a more in-depth discussion of why this happens? As the authors allude to, MAPPO-L has access to the global state during the centralized training phase, so it is okay (and probably expected) if the proposed method (which is fully distributed) underperforms MAPPO-L.
- Since a graph structure and distributed implementation are the underpinnings of the proposed method, how does the proposed method related to graph neural networks (GNNs)? The agents could perform message passing to their neighbors through a GNN architecture, so it would be helpful if the authors could discuss potential connections of their method to MARL methods based on GNNs, such as the following reference [A]:
[A] Jiang, Jiechuan, Chen Dun, Tiejun Huang, and Zongqing Lu. "Graph Convolutional Reinforcement Learning." In International Conference on Learning Representations. 2019.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
As mentioned under **Weaknesses**, the extension to non-tabular MARL settings is unclear. Furthermore, the robustness of the proposed method to imperfect communication among agents (e.g., noise perturbations, random communication link drops, delays, time/frequency resource constraints, etc.) needs to be studied.