Response of Submission 872 to Reviewer UzHb
We thank Reviewer UzHb for recognizing the significance, problem setting, theoretical concept, solution concept and evaluation of our paper. A detailed response is provided below.
> Q1: Self-containment and conciseness.
We appreciate this valuable feedback. To address this, we have:
* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.
* Included a concise proof sketch for each theorem to enhance self-containment.
* Reduced the use of mathematical notations and added more textual explanations.
Please refer to our global response and the revised manuscript for more details.
> Q2: Why do adversaries condition on observations despite them not being Markovian (in contrast to the non-adversarial policies)? Is this intentional or a mistake?
Thank you for pointing out the concern. Please allow us to clarify that in our original draft, for both defender and adversary, their policies are conditioned on histories $H$, not observations $o$. For example:
* Section. 2.1, paragraph 2, "...each agent $i$ selects its action $a_t^i \in \mathcal A^i$ using its policy $\pi^i(\cdot|H_i^t)$"; "the goal of all agents is to learn a joint policy ... that maximize the long-term return "$J(\mathbf{\pi}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0, \mathbf{a}_t \sim \pi(\cdot|H_t)\right]$".
* Section. 2.2, paragraph 2, "replaced by action $\hat{a}^i_t$ sampled from an adversary with policy $\hat{\pi}^i(\cdot|H_t^i, \theta)$"; "the value function can be defined as $V_\theta(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0=s, \mathbf{a}_t \sim \pi(\cdot|H_t), \hat{\mathbf{a}}_t \sim \hat{\pi}(\cdot|H_t, \theta)\right]$".
A thorough review of our paper confirms that the policy of both defenders and adversaries rely on histories, not observations.
> Q3: Do you have any intuition on why M3DDPG performs so poorly in the presence of an adversary compared to the standard MAPPO without any robustness mechanism in Figure 4?
There are two key reasons for M3DDPG’s underperformance in adversarial contexts, as compared to standard MAPPO without robustness mechanisms.
First, as discussed in Proposition 2.2, the existence of a pure-strategy equilibrium (RMPBE) isn't guaranteed. M3DDPG, which derives from MADDPG, inherently generates deterministic policies. Such policies may not be optimal in robust cooperative MARL contexts where a pure-strategy equilibrium is not assured. Despite this, M3DDPG was included as a baseline due to its recognition in the field.
Second, as a part of the max entropy RL family, MAPPO employs stochastic policies that encourage diverse solution strategies, thereby enhancing resilience to perturbations. This advantage of max entropy policies is well-documented in several studies [1, 2, 3, 4].
[1] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Ph.D. Thesis, Carnegie Mellon University, 2010.
[2] Haarnoja, Tuomas, et al. "Reinforcement learning with deep energy-based policies." ICML 2017.
[3] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." ICML 2018.
[4] Eysenbach, Benjamin, and Sergey Levine. "Maximum entropy rl (provably) solves some robust rl problems." ICLR 2022.
> Minors:
Thank you for your suggestions. We have meticulously revised the manuscript to correct grammatical errors and enhanced the clarity of Figure 2 in our revised submission.