We sincerely appreciate the reviewer's response and the insightful discussions. We seek to provide further clarification on this matter.
Primarily, as we elaborated in our previous rebuttal, we only imply that $\pi^*_{\theta, \phi}(a\mid s)\propto \pi_\theta(a\mid s)e^{Q_\phi(s, a)}$ is an optimal policy with respect to the **current estimated $Q_\phi(s, a)$**. This suggests that **$\pi^{\theta, \phi}(a\mid s)$ is not the global optimal policy** unless $Q_\phi(s, a)$ is the global optimal $Q$.
The correctness of this formulation can be substantiated by referring to Equations 7 and 8 in the MPO paper. Specifically, let's consider a typical RL algorithm that involves alternating between policy evaluation and policy improvement stages. In alignment with MPO's approach, during iteration $i$, the estimated Q value is denoted as $Q_{\theta_i}(s, a)$. Subsequently, within the policy improvement phase, the objective is to improve the current policy with respect to the $Q_{\theta_i}(s, a)$ values. Such a problem is formulated as a constratined optimization problem in MPO:
$$\max_q E_{\mu (s)}E_{q(a\mid s)}[Q_{\theta_i}(s, a)], \quad \mbox{s.t.}\quad E_{\mu(s)}[D_{KL}(q(a\mid s), \pi(a\mid s, \theta_i))]< \epsilon.$$
Here, we strictly follow the MPO's notation, where $\mu$ is the state distribution, $q(a\mid s)$ is a variational policy, i.e., the improved policy we aim to obtain, and $\pi(a\mid s, \theta_i)$ is the current policy. The above problem has the following closed-form solution by solving its Langrangian (refer to the Eqn. 8 of MPO):
$$q_i(a\mid s)\propto \pi(a\mid s, \theta_i)e^{Q_{\theta_i}(s, a) / \eta^*}$$
where $\eta^*$ is a normalizing factor obtained by solving another convex dual function. Note that throughout the derivations of MPO, only the estimated $Q_{\theta_i}(s, a)$ is involved, rather than the global optimal Q. The optimality claim in our paper and the previous rebuttal is actually with regards to the above constrained optimization. We believe this is sufficient to support the correctness of our derivations.
----
Also, we can derive this from the constrained policy search objective, following the AWR paper suggested by the reviewer. Referring to Section 3.1 in the AWR paper, our primary aim is to identify a policy that maximizes the expected improvement $\eta(\pi) = J(\pi) - J(\mu)$, where $\mu(a\mid s)$ is a sampling distribution. Instead of expanding $\eta (\pi) = A^\mu (s, a) = R_{s, a}^\mu - V^\mu (s)$ as in the AWR paper, let’s replace $R_{s, a}^\mu - V^\mu (s)$ with a function $f (s, a)$ for simplicity. In this case, we have our objective now as
$$E_{s\sim d_\pi(s)}E_{a\sim \pi (a\mid s)}[f (s, a)]$$
This equation suggests that our objective is to obtain **an optimal policy $\pi^{*}$, such that the $f(s, a)$ can be maximized under the expectation of** $E_{s\sim d_\pi(s)}E_{a\sim \pi (a\mid s)}[f (s, a)]$.
Next, becaues the above objective is hard to optimize due to the dependency between $d_\pi(s)$ and $\pi$, the AWR paper suggests to solve the following constrained policy search problem (Eqn. 5 and 6 in AWR).
$$\arg\max_\pi \int_s d_\mu(s)\int_a \pi(a\mid s)f(s, a)dads, \quad\mbox{s.t.}\quad \int_s d_\mu (s)D_{KL}(\pi (\cdot\mid s)\parallel \mu(\cdot\mid s)) \leq \epsilon$$
Furthermore, by solving the Lagrangian of this problem, we can derive a closed-form solution as presented in Eqn. 8 of the AWR paper:
$$\pi^*(a\mid s) = \frac{1}{Z(s)}\mu(a\mid s)e^{f(s, a)/\beta}$$
The equation above indicates that we have successfully derived a policy $\pi^*(a\mid s)$ that maximizes the expected value of the function $f(s, a)$ while satisfying the constraint $\int_s d_\mu (s)D_{KL}(\pi (\cdot\mid s)\parallel \mu(\cdot\mid s)) \leq \epsilon$. However, this derivation **does not impose any specific requirements on** $f(s, a)$ **as it solely involves solving the Lagrangian of a constrained optimization problem.** Consequently, if we choose $f(s, a) = Q_\phi(s, a)$, the estimated Q value function, this choice remains valid in a sense of maximizing the expected value of $Q_\phi(s, a)$ while satisfying the constraints.
----
We thank the reviewer again for the useful discussions and we will definitely improve our presentation of this statement in our revisions to avoid any potential confusions. Please kindly let us know if you have any further questions.