Summary
This paper proposes to train a mixing network and agent networks separately for multiagent reinforcement learning when the system is based on the centralized training and decentralized execution (CTDE) paradigm. The authors derive an individual reward that maintains invariance between the gradients of the mixing network's loss function and the sum of the loss functions of the agent networks. Then, the authors apply Prioritized Experience Replay to select samples for training the agent networks. The proposed method, DIFFER, is compared with QMIX, COMIX, QPLEX, and MASER on several tasks, such as SMAC, GRF, and MAMujoco. The experimental results show that DIFFER outperforms the baseline methods and demonstrate that a fair experience replay mechanism works efficiently.
Strengths
- Originality: The idea of fair experience replay based on the constraint between the gradients of the mixing and the agent networks is novel. However, I found the following paper that may be related to this study: D. Hostallero et al. (2020). Inducing Cooperation through Reward Reshaping based on Peer Evaluations in Deep Multi-Agent Reinforcement Learning. In Proc. of AAMAS.
- Quality: The experimental results support the claims and the proposed method.
- Clarity: The manuscript is written very well and easy to follow.
- Significance: While the CTDE approach becomes popular in MARL, many previous studies do not pay attention to the difference between individual experiences. This study sheds light on its importance.
Weaknesses
- Although the experimental results show that the proposed method works well, there is no theoretical analysis. In particular, the gradient of the loss function of the agent network is independent of the target networks as described later. It would be better to discuss the loss function in detail.
- In addition, the off-policyness of the proposed method should be discussed.
Questions
Major comments:
- The individual reward (2) is interesting, but I am unsure how the TD error is calculated. I think that the TD error is given by
$$
(R + \gamma \tilde{Q}_{\mathrm{tot}} - Q_{\mathrm{tot}}) \frac{\partial Q_{\mathrm{tot}}}{\partial Q_i}
$$
because the term $\gamma \tilde{Q}_i – Q_i$ cancels out. Is the target network $\tilde{Q}_i$ necessary? Is my understanding correct?
- I would like to know whether the used learning algorithm is off-policy or not. Experience replay buffer technique is typically applied to off-policy reinforcement learning algorithms, but the loss functions (6) and (7) suggest that the action value function is trained in an on-policy manner, like SARSA. Is on-policyness necessary for the proposed method?
- Is the input of the mixing network consistent with the outputs of the agent networks when the mixing and agent networks are trained separately?
- Does QMIX-divide adopt uniform sampling instead PER for training the agent networks? If so, the performance difference between QMIX-DIFFER and QMIX-divide can be discussed based on Fujimoto et al. (2020).
S. Fujimoto et al. (2020). An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay. NeurIPS 33.
Minor comments:
- Line 87: $A^n$ -> $A^N$.
- Line 96: $r$ -> $R$.
- Line 154: Since $S$ denotes the state space, it would be better to use a different symbol.
- Eq.(7) and Line 191: $\chi^{\mathrm{indi}}_j$ -> $\chi^{\mathrm{ind}}_j$.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
Minor comments:
The authors do not discuss this work's potential negative social impacts.