Summary
The paper introduces a robust Multi-Agent Reinforcement Learning (MARL) framework called ERNIE (adversarially Regularized multi-ageNt reInforcement lEarning). The goal of ERNIE is to address the issue of policy robustness in MARL algorithms. The key contributions of the paper are as follows:
1. Robust MARL Framework: The authors propose ERNIE, which promotes the Lipschitz continuity of policies through adversarial regularization. This regularization enhances robustness against noisy observations, changing transition dynamics, and malicious actions of agents.
2. Lipschitz Control: ERNIE controls the Lipschitz constant of policies, allowing them to be more resilient to variations in the environment. This control is achieved by leveraging adversarial regularization, which encourages Lipschitz continuity with respect to state observations and actions.
3. Training Stability: The paper addresses the training instability that adversarial regularization can introduce by reformulating it as a Stackelberg game. This reformulation provides a more stable training process, making the optimization problem smoother for the defender (policy).
4. Experimental Evaluation: The authors extensively evaluate the effectiveness of ERNIE in traffic light control and particle environments. The experiments demonstrate that ERNIE improves robustness and performance compared to non-robust MARL approaches.
5. Extension to Mean-field MARL: ERNIE is extended to mean-field MARL, a strategy to scale up MARL algorithms.
6. Theoretical Insights: The paper provides theoretical insights into the smoothness and robustness of policies in MARL. It shows that in environments with smooth transition and reward functions, policies' value functions are also smooth. The authors prove the existence of smooth and close-to-optimal policies in such environments and establish that policy robustness is proportional to its Lipschitz constant.
Strengths
**Originality**: The theoretical insights into the smoothness of environments and robustness of policies look interesting to me. By assuming the smoothness of environments, we can get some good properties of robustness. This idea is very interesting to me. However, this idea is very similar to the paper [1]“****Deep Reinforcement Learning with Robust and Smooth Policy****”.
[1] Shen, Q., Li, Y., Jiang, H., Wang, Z., & Zhao, T. (2020, November). Deep reinforcement learning with robust and smooth policy. In *International Conference on Machine Learning* (pp. 8707-8718). PMLR.
**Quality**: The paper is very informative. I can see the authors have a lot to show readers in the paper: the theoretical results, the proposed framework, how to solve the problem efficiently, and how to adopt the proposed framework to large-scale multi-agent systems.
**Clarity**: The paper appears to be clear in its presentation of the problem, the proposed solution, and the results of the experiments. The authors provide a theoretical motivation for their algorithm, explaining how Lipschitzness (smoothness) serves as a natural way to gain robustness while reducing the policy search space. The extension of ERNIE to mean-field MARL is also clearly explained.
**Significance**: I believe this paper's theoretical analysis of how smoothness helps in training robust policies would help future researchers in the field of robust MARL to develop better solutions to the robustness issues.
Weaknesses
**Confusing Writing**: I found the paper sometimes inconsistent in motivation, methods, and experiments, which makes me confused about what problem this paper tries to solve. In the abstract, the authors point out that we need to consider real-to-sim gaps in the deployment of MARL algorithms, I quote the original sentence there “This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment.” But later, the authors claim that their proposed method is robust to perturbations on observations and actions. I also quote the original sentences as follows: “Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization.” and “In particular, we develop an adversarial regularizer to minimize the discrepancy between each policy’s output given a perturbed observation and a non- perturbed observation.” When I dive into the methodology of this paper, I found most of the contexts are about **Robustness against observation noise with smooth policies** and **Robustness against Malicious Actions.** We can see that the abstract and some texts in the introduction mislead readers about what kind of robustness this paper tries to achieve in MARL.
**Evaluations on Simple Games**: The evaluation of the proposed methods is conducted on specific tasks such as traffic light control and particle environment tasks. While these tasks are complex, they may not fully reveal insights into how the smoothness of environments affects the robustness of policies. I would suggest running experiments on some simple matrix games or small-scale multi-agent toy examples [1] to better study how smoothness is connected to robustness.
[1] Hu J, Wellman MP. Nash Q-learning for general-sum stochastic games. Journal of machine learning research. 2003;4(Nov):1039-69.
**Lack of Comparison with Robust MARL Methods:** This paper claims that the proposed method is robust to perturbed actions and observations. Considering there are some existing works about robust MARL considering adversarial-purebred observations [1, 2, 3] and perturbed actions [4], should the authors compare the proposed method with at least one of these robust MARL methods?
[1] Han S, Su S, He S, Han S, Yang H, Miao F. What is the solution for state adversarial multi-agent reinforcement learning?. arXiv preprint arXiv:2212.02705. 2022 Dec 6.
[2] He S, Han S, Su S, Han S, Zou S, Miao F. Robust Multi-Agent Reinforcement Learning with State Uncertainty. Transactions on Machine Learning Research. 2023 Mar 10.
[3] Zhou Z, Liu G, Zhou M. A Robust Mean-Field Actor-Critic Reinforcement Learning Against Adversarial Perturbations on Agent States. IEEE Transactions on Neural Networks and Learning Systems. 2023 Jun 5.
[4] Li, S., Guo, J., Xiu, J., Yu, X., Wang, J., Liu, A., ... & Liu, X. (2023). Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game. *arXiv preprint arXiv:2305.12872*.
**Writing Suggestions:**
The mean-field MARL part does not tightly connect to the main text of this paper. Would it be better to make it a remark instead of a subsection? Besides, the organization of this paper makes me confused, for example, Robustness against observation noise with smooth policies (a paragraph) and Robustness against Malicious Actions (a subsection) are placed on different layers of this paper. Should they be on similar layers? As a paper studying robust MARL, in the related work, this paper uses more text to discuss robust RL works than robust MARL. The literature review of robust MARL is not comprehensive.
Questions
It would be great if the authors can resolve my concerns in the weakness box and the Originality concerns in the strength box. I will decide my final score after the rebuttal phase according to the authors' responses.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.