Reply to the comment of Reviewer rt2G
Thank you for your thorough proofreading! We apologize for the oversight in explaining certain symbols, and we're committed to addressing this issue and ensuring their proper clarification. Your feedback is greatly appreciated, and we'll strive to make the necessary improvements.
**1. The distinctions among $\bar{Q}$, $\hat{Q}$, and $Q$ are not explained. The definition of $\bar{\epsilon}$ in Eq.(4) is omitted.**
In principle, "\hat" refers to the symbols involved in MACQL, like the empirical $Q$ function, the empirical Bellman operator, the empirical TD error, etc. And "\bar" refers to the corresponding symbols involved in CFCQL.
Concretely, we employ $\hat{Q}$ to represent the empirical Q function after each iteration in MACQL, and the $\bar{Q}$ to represent the empirical Q function after each iteration in CFCQL. Similarly, $\bar{\mathcal{E}}$ serves a comparable role, signifying the empirical TD error in the loss of CFCQL, analogous to the function of $\hat{\mathcal{E}}$ in MACQL.
Regarding $Q$, it usually represents a variable that need to be optimized (Eq. 1,2,3,4,7), or the actual $Q$ function obtained via exact policy evaluation(Theorem 4.1), distinguished from the empirical $Q$. We will give detailed explanation about the symbol in the revised manuscript.
**2. What's the difference between the two operators in Algorithm 1, Line 6?**
These are two Bellman operators suitable for discrete and continuous action space, respectively. The main difference is that $\mathcal{T}$ uses the IGM principle to find the actions with max $Q^{tot}$ in the next time step, and $\mathcal{T}^{\pi}$ uses a parameterized policy to output the actions of the next time step. For detailed explanation, please refer to Section 3.2 "Value Functions in MARL".
**3. The term "Ablation study" in Figure 3 seems misplaced.**
Thank you for bringing this to our attention. We have made the necessary correction, now stating "hyperparameters examination."
**4. The rationale for introducing the parameter $\lambda$.**
The rationale behind parameter $\lambda$ is rooted in the understanding that uniformly penalizing the out-of-distribution (OOD) actions of each agent might not be optimal, considering that the OOD degrees across agents may vary. It's important to note that Theorem 4.2 necessitates only $\sum_i \lambda_i = 1$. Consequently, we introduce parameter $\lambda$ to impose higher penalties to agents displaying more significant deviations from the dataset. This is achieved under the constraint $\sum_i \lambda_i = 1$, ensuring that value function overestimation is avoided.
**5. A clear explanation of what "counterfactual Q-learning" means.**
In MARL, counterfactual refers to considering the return of $a^i$ during the training process, taking into account the situations where other agents take different actions. Therefore, the marginal distribution of $Q$ value without $a^i$ is obtained by taking the expectation of different actions of other agents，and it only measures the impact of action $a^i$ on the $Q$ value[1]. In our context, when we pessimistically evaluate the $Q$ value for agent $i$, we keep the other agents' actions sampled from the dataset , which is equivalent to only considering the pessimistic impact of $a^i$ on $Q$ values. Therefore, our approach shares similar principles with the counterfactual method, so we adopted this term to refer to our method.
**6.The relations between $\pi$, $\mu$ and $\beta$. Why is the update of $\beta^{-i}$ not included in Algorithm 1?**
First, let us clarify the significance of $\beta$. $\beta$ denotes an implicit distribution over the dataset $D$, signifying that the dataset can be considered as being collected by executing the behavior policy $\beta$ in the environment. Consequently, $\beta$ remains fixed and can not be updated. On the other hand, $\mu$ primarily emerges within the theoretical analysis of policy evaluation, and it can denote any policy different from the behavior policy $\beta$. The notation $\pi$ pertains to the learning policy. In practical algorithms, we often opt for $\mu=\pi$ for simplicity's sake, which is also consistent with the treatment in CQL[2]. For ease of understanding, you can equate $\mu$ and $\pi$, treating them as the same – that is, referring to the learning policy. This clarification will be accentuated in the revised manuscript.
Reference:
[1] Foerster J, Farquhar G, Afouras T, et al. Counterfactual multi-agent policy gradients[C]//Proceedings of the AAAI conference on artificial intelligence. 2018, 32(1).
[2] Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 1179-1191.