Thank you for your valuable feedback and suggestions!
We answer the raised concerns point by point below. We will write $\mathbb{E}$ as $E$ due to markdown formatting problems.
1. In the MFG imitation learning setting with behavior cloning, the agent seeks to learn a policy $\pi^{A}$ that minimizes the following expected KL-divergence objective: $\epsilon^{BC} := \max_{n\in[H-1]} \epsilon_n^{BC}$, where $\epsilon_n^{BC}:=E_{s \sim \rho_{n}^{E}}[\text{KL}(\pi_{n}^{E}(\cdot|s)||\pi_{n}^{A}(\cdot|s))]$. Although the agent may assign a zero probability to an action that the expert assigns a strictly positive probability, in this case the $\epsilon^{BC}$ equals infinity.
However, this scenario is quite unlikely (e.g., if the agent policy is a softmax, a classic approach).
Furthermore, using Pinsker's inequality, it can be shown that $E_{s\sim\rho_{n}^{E}}[\pi_{n}^{E}(\cdot|s)-\pi_{n}^{A}(\cdot|s)]\leq \sqrt{2E_{s\sim\rho_{n}^{E}}[\text{KL}(\pi_{n}^{E}(\cdot|s)||\pi_{n}^{A}(\cdot|s))]}\leq\sqrt{2\epsilon}.$ It is crucial to emphasize that substituting the $\ell_1$ BC bound in Eq.3 with the KL-divergence bound above, yields the desired BC error with $\sqrt{\epsilon}$ dependence.
More precisely, apply the above inequality to line 689 to update Lemma 3 and to line 701 for Lemma 4.
As an immediate corollary (showing an exponential gap between single agent and mean field games in imitation learning settings), the Nash imitation gap satisfies $\mathcal{E}(\pi^{A})\leq H^{2}(r_{\max}+2L_{r})\sqrt{2\epsilon^{BC}}$ if $L_P=0$ (Corrolary of Theorem 1), else $\mathcal{E}(\pi^{A})\leq\Big(\frac{2(L_{r}+r_{\max})}{L_{P}^{2}}(1+L_{P})^{H}+H^{2}r_{\max}\Big)\sqrt{2\epsilon^{BC}}$ when $L_P>0$ (Corrolary of Theorem 3).
We can easily derive similar results for the adversarial setting by considering the JS divergence optimized by GAIL.
Moreover, similar to the infinite-horizon single-agent IRL setting, studied by Xu et al. 2020 [1], our analysis does not require any additional assumptions.
Following the reviewer's comments, we will add this explanation more clearly in the next version of the paper.
2. We agree with the reviewer that without access to the transition kernel, there will be errors in estimating the population distribution $\mu^{\pi}$, resulting in errors in the final recovered policy.
However, notice that very few practical approaches attempt to estimate occupancy measures.
Rather, an adversarial approach avoids this estimation.
We explain in Section 4.3 and Appendix A how adversarial approaches could be extended to our setting by replacing the inner RL loop with an inner MFC loop, thus avoiding the need to estimate this kind of distribution.
Moreover, it is crucial to emphasize that our main contribution is to show a gap between single-agent IL and MFG IL, i.e., the exponential dependence on the horizon of traditional IL methods (and also of state-of-the-art algorithms for MFG IL [2,3,4]).
More specifically, we provide a systematic analysis of the MFG imitation learning setting and show which type of distance needs to be optimized, i.e., the MFC-ADV, which is different from those used in the literature.
In addition, we provide a new experiment that empirically shows the exponential dependence of BC and vanilla-ADV on the horizon (see the rebuttal pdf file).
While designing a practical algorithm remains an important open question, it is important to emphasize that addressing it is beyond the scope of this paper.
Nevertheless, we think that the difficult part is how to deal with the MFC loop, rather than estimating the population distribution, which can be avoided thanks to an adversarial approach.
3. Thank you for raising this important point.
Consider the set of populations $\rho_n$ as defined in Definition 2, induced by all stochastic policies.
A sufficient condition for swapping the min and max is that this set is convex, i.e., that the transition kernel is such that, for any $\rho^1_n$ and $\rho^2_n$ in this set, then $(1-\alpha) \rho^1_n + \alpha \rho^2_n$ is also in this set, for any $\alpha\in (0,1)$.
For example, this is satisfied if the transition kernel is linear in the population.
It is also satisfied by the MFG we consider in Appendix C (for which the transition kernel is not linear in the population).
A full characterization of transition kernels that satisfy this property is beyond the scope of this paper.
Notice that this is not required for the stated theorems, only for the sketched practical adversarial approach.
We would also like to underline that our main contribution is to show an exponential gap between single agent and mean-field games in the imitation learning settings.