Natural gradients
We thank the reviewer for having read our rebuttal and for the timely feedback. Below our reply.
1. We agree with the reviewer that our Introduction section may be improved by revising previous works using the KL divergence in policy optimization, along similar lines as [30, 32, 36], so that our motivation of using the Wasserstein distance is more clear in the context of "regularized objectives" in policy optimization. We plan to include the following short revision in Section 1 (from line 49) of the revised version of the paper (including some minor text changes):
"A well-established technique for enhancing the stability of policy optimization involves introducing regularization to the objective function using the Kullback-Leibler (KL) divergence [40, R1, R2]. This regularization mechanism aims at maintaining small changes between successive policy updates. Policies similarity can also be quantified via the Wasserstein distance, as recently proposed in [30, 32, 36]. Unlike the KL divergence, the Wasserstein distance enjoys powerful geometrical, computational, and statistical features [26, 27]. In this paper we exploit such properties in a Wasserstein-regularized objective for GMM-based policies optimization. This allows us not only to see the policy iteration as a Wasserstein gradient flow, as in [30], but also to leverage the Riemannian geometry associated to the GMM space to formulate exact Riemannian gradient descent updates."
Moreover, we will provide further details in Appendix A.4 so that the connections among our implicit scheme, the approximation with NGs, and the retraction operation in Riemannian manifods become clearer.
We believe that the minor changes proposed above complement and improve the motivation and position of our method w.r.t KL-regularized approaches.
[R1] Towards an Understanding of Default Policies in Multitask Policy Optimization. AISTATS, 2022.
[R2] Information Asymmetry in KL-regularized RL. ICLR, 2019.
2. We would like to point out that we did not deliberately omit an empirical comparison against NG- or KL-based approaches, and we did not intend our paper to "look suspicious", as this reviewer suggests. In this regard, we would like to highlight that:
- Our original benchmark experiments included PPO, which optimizes a KL-regularized objective [44, Eq. 8].
- During our rebuttal, we added a GMM-based formulation of PPO, i.e, PMOE-PPO.
- Our ablation study reported in App. A.6.2 is aimed at showing experimental evidence on the difference between our Riemannian approach and the implicit scheme. We would like to emphasize that the natural gradient is an approximation of such an implicit scheme, as discussed in our Appendix and in other related works.
Although we did not compare against an NG-based approximation of our method, we still believe that the results reported in our paper and the rebuttal provide empirical evidence of our approach performance against methods that include KL divergence (as soft constraints), and an implicit schema (related to the inexact NG). So, we find the reviewer's argument "the complete omission of NG and the KL looks suspicious" inaccurate and unfair.
3. Note that a method that formulates the Wasserstein NG for multivariate GMMs in policy optimization has not been proposed yet (to the best of our knowledge). As pointed out in our previous answer, Moskovitz et al [36] work is the only work where Wasserstein NG is used in policy optimization, but this does not propose a formulation for GMM policies. Therefore, in order to carry out such comparison we would need to develop a new policy optimization approach that builds on the use of the Wasserstein NG developed in [R3], and that employs similar numerical estimates as proposed in [36], perhaps independently for each component of the GMM. Still, such an approach would disregard the positive-definite constraint of the Riemannian manifold arising from the covariance matrices (for which approximated projections would be then necessary). As a conclusion, we agree with the reviewer that such an approach would be relevant, but it is non-existent in the current literature. We believe this could be an interesting future research direction, although it remains an approximation of our mathematically rigorous method.
[R3] Optimal transport natural gradient for statistical manifolds with continuous sample space, Information Geometry, 2020.