Actor-critic (AC) methods are widely used in reinforcement learning (RL) and benefit from the flexibility of using any policy gradient method as the actor and value-based method as the critic. The critic is usually trained by minimizing the TD error, an objective that is potentially decorrelated with the true goal of achieving a high reward with the actor. We address this mismatch by designing a joint objective for training the actor and critic in a decision-aware fashion. We use the proposed objective to design a generic, AC algorithm that can easily handle any function approximation. We explicitly characterize the conditions under which the resulting algorithm guarantees monotonic policy improvement, regardless of the choice of the policy and critic parameterization. Instantiating the generic algorithm results in an actor that involves maximizing a sequence of surrogate functions (similar to TRPO, PPO) and a critic that involves minimizing a closely connected objective. Using simple bandit examples, we provably establish the benefit of the proposed critic objective over the standard squared error. Finally, we empirically demonstrate the benefit of our decision-aware actor-critic framework on simple RL problems.
Paper
Similar papers
Peer review
Summary
The authors present a decision aware actor-critic algorithm and then try to analyse the same. Subsequently they also provide some numerical results.
Strengths
The authors try and provide an analysis of an actor-critic algorithm. However, there are a number of questions that are unclear to me that I write in detail below.
Weaknesses
I am writing down the entire list of comments on the paper below. One can call these weaknesses as these points have not been explained well or there are issues in the explanations provided. 1. The authors say in the introduction that the objective of the critic is to minimize the value estimation error across all states and actions. This is incorrect since a critic's job is to estimate the expected TD error for any given policy provided by the actor and the actor's job is to find the policy that minimizes this expected TD error over policies. The authors seem to mention this at multiple places. 2. Weird notation: J_s(\pi) is well defined but what is J_s(\rho)? Note that \rho is the initial state distribution and then you say J(\pi) = E[J_s(\rho)]! 3. Note that d^\pi and \mu^\pi are not valid probability distributions. How do you then value sample states or state-action tuples from these? 4. What does it mean to say that since p^\pi(s,a) is a probability, you can write it in the Boltzmann form as you have written. Is it always possible to express any probability distribution as some Boltzmann distribution? Later you introduce a parameterization \theta to this distributional form. Why not directly give a parameterized Boltzmann probability if that is what you want for the ramdomized policy instead of doing this in such a round about manner. 5. In the definition of \pi_{t+1} in Section 3.1, obviously \eta plays a major role. What values can it take? 6. For the blue term on page 4, you mention that as c decreases, critic error decreases. I don't see how? As c decreases, the first term inside D_{\Phi^*}(.,.) will decrease as c decreases but that term is multiplied by 1/c which will blow up as c decreases. 7. Algorithm 1: How do you compute $\nabla_v L_t (v_k) and \hat{g}_t? 8. Step-sizes and stability of the algorithm: You use \alpha_c and \alpha_a as the step-sizes for the critic and actor respectively. I don't see any conditions written on these step-sizes. The reason is that actor-critic algorithms are meant to track policy iteration. This requires that the critic recursion moves on a faster timescale as compared to the actor recursion. When diminishing step-sizes are used, one requires that $\alpha_a go to zero at a rate faster than \alpha_c in order to get this effect of asymptotic convergence. Furthermore, the authors do not mention anything about the stability of their procedure. How does one ensure that this algorithm is stable - a precondition to ensuring convergence. 9. Since you do not assume anything about mixing times of the underlying Markov process under any policy, how do you ensure that Markov noise does not create any problems in convergence of the scheme, see for instance,Sajad Khodadadian, Zaiwei Chen, Siva Theja Maguluri, "Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm," ICML 2021. 10. In Proposition 2, what are the terms \tilde{H}_t^\dagger and [\hat{g}_t]_{s,a}? 11. How does one interpret Proposition 3? Not clear at what rate the second term decreases and whether it even does so? In the absence of a proof of decrease and the rate at which it happens, the result is meaningless. 12. In Section 4, you give a monotonic policy improvement result. TRPO also has a similar result. Which one is better - the improvement provided by your algorithm or TRPO? 13. Mistakes: Section 5.1 - you say \nabla_\pi J(\pi) = d^\pi(s) Q^\pi(s,a). What is s and what is (s,a) on the RHS since the LHS does not have any s or (s,a) tuple? Also, in Section 5.2, you say \nabla_\pi(J(\pi)) = d^\pi(s) A^\pi(s,a) p^\pi(a|s)? What are s, a on RHS. There is clearly a mistake since LHS does not depend on these quantities. Also, how do you reconcile these two definitions for the same object? 14. The bandit proposition 5 and also 7 suddenly come in and appear crude with such assumptions as deterministic rewards, deterministic Q-value updates etc. It is not clear why these results have been provided.
Questions
I have given my questions in detail above.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
I think the analysis of convergence is flawed since there are no assumptions or results regarding (a) Lipschitz continuity of the objectives, (b) step-sizes used in the actor-critic scheme, (c) stability of the two coupled recursions, (d) fast mixing nature of the Markov noise, etc., have been made or shown. Moreover, from my comments mentioned above, the paper lacks on several fronts and will require significant revision and re-review in order to be publishable.
Summary
The paper addresses the issue of objective mismatch in Actor-Critic methods by designing a joint objective that enables training the actor and critic in a decision-aware manner. The proposed algorithm ensures monotonic policy improvement, irrespective of the chosen policy and critic parameterization.
Strengths
The paper presents an algorithm that guarantees monotonic policy improvement and has solid theoretical foundations.
Weaknesses
The experimentation in the paper is somewhat limited, which may raise concerns about the algorithm's performance in more complex environments.
Questions
In complex real-world environments, $\Delta_\pi J(\pi)$could only be estimated, e.g. using MC sampling. And for large MDPs, function approximation error of Q is unavoidable for large MDPs. The article provides experiments on two simple environments, Cliff World and Frozen Lake, using linear/tabular action and Linear cirtic. Thus, my questions are: (1) Can the algorithm be applied to more challenging environments, such as the standard benchmark tasks in MuJoCo? If so, how well does it perform in those environments? If there are limitations or challenges in applying the algorithm to more difficult tasks, it would be helpful to elaborate on those limitations. (2) Does the algorithm introduce significant extra time overhead compared to other methods? It would be useful to report the additional time required by the algorithm on the Cliff World and Frozen Lake environments.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Potential Challenges in Complex Environments: The applicability of the proposed algorithm to more complex environments, such as those found in the MuJoCo benchmark tasks, remains uncertain.
Thank the authors for their rebuttal. I don’t carefully review your theorems, as such, my perspective on that particular aspect is limited. However, theory and experimentation are not conflicting endeavors. Therefore, I recommend the authors to include more compelling experiments to enhance the quality of the work. This is not beyond the scope of your work. I respond quickly because I hope that perhaps during this discussion period, you could attempt some experiments using MuJoCo, as continuous control problems are crucial application scenarios. I'm not asking for results in all scenarios of MuJoCo; providing results in one or two scenarios would suffice. Furthermore, regarding the time complexity here, I'd like to know the wall time clock data if possible. Because using a different loss function might come with additional costs. Of course, those mentioned above are just some suggestions on my part. If you're unable to provide them, it's not a significant issue
Experimental evaluation
[1] *I hope that perhaps during this discussion period, you could attempt some experiments using MuJoCo* We thank the reviewer for engaging with the rebuttal. We agree that theory and experimentation are not conflicting endeavors, and consequently, we did provide experimental evidence that validates our theoretical results. As briefly alluded to in the rebuttal, there are several reasons why we did not consider Mujoco experiments in the current paper: 1. Mujoco experiments typically require over-parameterized deep neural networks for the critic. Since these models are highly expressive and can interpolate the data [a], it is possible to estimate all state-action values for simpler environments in the Mujoco suite. In this case, the choice of the critic loss becomes irrelevant. This effect can already be seen in our experiments -- In Figure 1, for $d = 80$ (corresponding to using a highly expressive model), the choice of the critic loss does not matter and all methods have similar performance. 2. For more complex environments, the experimental results for Mujoco are significantly affected by the choice of hyper-parameters [b], and secondary implementation-level factors have a major impact on the algorithm performance [c]. Unless we do careful ablations controlling for these factors, these experiments will not effectively test the paper's contribution. If the reviewer can suggest an experimental protocol that could help isolate the effect of the critic loss, we would be grateful and could test it. 3. Finally, our work is in line with similar theoretical papers that provide the necessary framework for developing algorithms with performance guarantees. Aware of the massive amount of engineering and finetuning required to develop state-of-the-art RL methods from these theoretical frameworks, we felt that it would be more honest to restrict our experiments to evaluate the specific advantages offered by our approach, highlighting the issues of standard approaches, without claiming to have developed a full-fledged RL algorithm. That being said, your point is well taken. We agree that more experiments will indeed enhance the quality of our work. Consequently, we are currently running experiments on Cart Pole, one of the simpler continuous control environments. Specifically, we are considering a linear critic with tile-coded features (following the experimental protocol in [d]). By varying the dimension of these features, we hope to study the effect of the critic loss. Because of time and computational constraints, we are not sure if we will be able to finish this set of experiments (with proper ablations) before the discussion period finishes. We will definitely add these experiments to the final version of the paper. [a]. Zhang et al, "Understanding deep learning requires rethinking generalization", 2016 [b]. Henderson et al, Deep reinforcement learning that matters, 2018 [c]. Engstrom et al, "Implementation matters in deep policy gradients: a case study on PPO and TRPO", 2020 [d]. Jain et al, "Towards Painless Policy Optimization for Constrained MDPs", 2022 2. *Furthermore, regarding the time complexity here, I'd like to know the wall time clock data if possible. Because using a different loss function might come with additional costs.* All methods have similar wall clock times. In particular, we report the average (across $1000$ outer iterations of Alg 1) time taken to minimize the critic loss corresponding to each method (TD, Adv-TD, and Ours) for the linear critic parameterization (with $d = 60$). Specifically, we use gradient descent with Armijo line-search (to automatically set the step-size $\alpha_c$) and terminate the algorithm when the gradient norm decreases below $10^{-6}$. a. On the Cliff World environment, the wall clock times for the Decision-aware, TD, and Adv-TD methods are $0.0550$, $0.0779$, and $0.0427$ respectively. b. On the Frozen Lake environment, the wall clock times for the Decision-aware, TD, and Adv-TD methods are $0.0020$, $0.0051$, and $0.0017$ respectively.
Suggestions on implementations.
Thank you so much! I think conducting experiments on CartPole is a fantastic initiative. I agree that the critical loss might attenuate the impact of your design, and my suggestion is that perhaps instead of comparing it directly with baselines, you could consider conducting an ablation study. Another suggestion is that you might explore implementing your critic choice concept directly on SAC, which could be a potential avenue for future work. I don't have any further questions, and once again, I truly appreciate your efforts.
Summary
The authors develop a generic decision-award AC algorithm where both the actor and the critic take steps iteratively to optimize some “policy improvement lower bound” under the FMAPG framework. In essence, each step takes the gradient estimation error as the critic error, and characterizes the policy improvement in terms of the estimated gradient $\hat g$. This result seems to come from the Taylor expansions of the Bregman divergence. The authors argue that since both the actor and the critic act in a cooperative manner, the critic therefore only focuses on parts of the state-action pairs that have the largest impact on the actor’s performance improvement. The authors provide the conditions for monotonic policy improvement and show that the algorithms converge to some stable points. Examples and experiments show that the proposed algorithm outperforms TD/AdvTD methods on certain tasks.
Strengths
The algorithm is interesting and well-motivated. Although the idea is not new to the RL community, the theoretical results in this work are sound and in general support the authors’ claims. Moreover, the paper is well written and I find most of the statements clearly presented.
Weaknesses
In order to perform GD/SGD, you at least need to rollout trajectories collected under $d^{\pi_t}$. I didn’t find evidence supporting the sample efficiency of the proposed algorithm as you have claimed. Questions include: 1. How many samples are used for each GD step in your experiment? 2. How large is the variance of the proposed algorithm compared to TD/AdvTD when the sampling budget is limited, especially when the samples do not have sufficient coverage? 4. How do the relative scales of $m_c$ and $m_a$ influence your results?
Questions
1. In practice, buffers containing trajectories collected under history policies are commonly used. Is there any evidence showing that your algorithm can handle the distribution shift issue? 2. Can you perform gradient steps for the actor/critic simultaneously? If so, what are the relative step sizes for the inner and the outer loops?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
No concerns here.
Summary
This research addresses the mismatched objectives in actor-critic (AC) methods used in reinforcement learning (RL). By introducing a joint objective for training the actor and critic in a decision-aware fashion, a generic AC algorithm is developed. The algorithm ensures monotonic policy improvement regardless of the policy and critic parameterization. The proposed approach offers advantages over traditional methods, as demonstrated through rigorous analysis and empirical evaluations.
Strengths
I think this work really pushes the RL community research efforts further by answering: > can we design a generic actor critic algorithm with joint objective? The main contribution of generic actor critic algorithm with joint objective is a really nice idea worthy for publication at NeurIPS.
Weaknesses
I have only have one minor weakness for this work as follows: > Using linear representations as "general function approximation" is a bit weak. I presume this is the reason why only simple RL problems have been demonstrated in this work. I am open to discussions with the authors and reviewers to increase my score. All the best for future decisions!
Questions
na
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
4 excellent
Limitations
na
Are you satisfied by the answers?
Dear reviewer, Would you please indicate whether the authors' response is satisfactory for you? If not, please engage with the authors, so we can get a better assessment of this work. Thank you, Area Chair
response to rebuttal to uLAb
I appreciate the authors responses. However, many of my comments have been ignored. The most important of these is of stability of the procedure? 1. Since the resulting scheme is a stochastic approximation algorithm, proving stability of such a procedure is important and there is no result in the paper which talks about stability. In other words, how do you ensure that the resulting stochastic iterates remain uniformly bounded almost surely on any sample trajectory? 2. Also, the response concerning the choice of step-size is not convincing enough from a theoretical point of view. I say this because in actor-critic algorithms such as those of Konda and Borkar, SIAM J. Control and Optimization (1999) or Konda and Tsitsiklis, SIAM J. Control and Optimization (2003), it is assumed that the critic step-size converges to zero slower than the actor's. 3. Such an argument is missing here as I also don't see how the noise is being treated here? Is it a martingale difference sequence or is it also Markovian? It should be Markovian because you are looking at samples coming from a Markov process. The convergence of the noisy scheme is not clear from the arguments. These are important concerns that cannot simply be wished away. I am not convinced with the responses to my questions and so I shall retain my rating of 3.
Further clarifications
We thank the reviewer for engaging with the rebuttal. We believe that we have addressed all of the reviewer's comments, and that there is still some misunderstanding. [1] *...Since the resulting scheme is a stochastic approximation algorithm...* As we have explained in the rebuttal (see the global response to all reviewers), **Alg. 1 is not a stochastic approximation algorithm similar to [Konda and Borkar, 1999]**. In order to clarify the difference between Alg 1. and the two time-scale setting in [Konda and Borkar, 1999], [59] (in our references), we can think of actor-critic as solving a bi-level optimization problem. Since the actor uses the $Q^\pi, A^\pi$ estimates from the critic in order to compute its loss and update the policy, the outer-level objective corresponds to the actor loss, whereas the inner-level objective is the critic loss. Similar to [23,60, a,b], Alg 1. aims to solve the inner-level optimization problem *using multiple critic updates* (Lines 5-8). On the other hand, the two time-scale algorithm in [59] performs one step of gradient descent (critic update) on the inner-level objective, followed by one gradient ascent step (actor update) on the outer-level objective. [2] *...there is no result in the paper which talks about stability...* Even though Alg 1. involves multiple critic updates, it is not required to *exactly* minimize the critic loss at iteration $t$. Specifically, the advantage of using the proposed joint objective for the actor and critic is that Proposition 2 guarantees monotonic improvement in $J(\pi)$ as long as the critic loss is less than a certain threshold. This result only depends on the magnitude of the critic loss (after $m_c$ updates). The form of the critic update (including the step-size) required to achieve that loss is irrelevant. Importantly, **the actor and critic are coupled via the threshold that the critic loss needs to achieve in order to guarantee policy improvement. Specifically, this threshold depends on the norm of the functional policy gradient (see Line 218)**. Hence, if this threshold on the critic loss is satisfied, Alg 1. can result in convergence to a stationary point of $J(\pi)$. On the other hand, if the critic loss does not satisfy such a threshold, Proposition 3 still guarantees the convergence of Alg 1. to the neighborhood of a stationary point i.e. for the special case of the Euclidean mirror map, we can obtain a policy $\bar{\pi}\_T$ s.t that $||\nabla J(\bar{\pi}\_T)||^2 \leq O(\frac{1}{T} + \epsilon_{\text{critic}})$ where $\epsilon_{\text{critic}}$ is the critic error. On the other hand, in order to prove guarantees, the analysis in [59] requires reasoning about both the form and step-size of the critic update. **Importantly, in [59], the actor and critic are coupled through the step-size. Specifically, the critic step-size converges to zero at a slower rate than the actor's, effectively enabling more critic updates** Unlike conventional actor-critic analyses including [Konda and Borkar, 1999], [59], our theoretical results hold for non-linear function approximation and can handle off-policy updates. **Hence, our theoretical guarantees are much stronger than the stability results that the reviewer is alluding to.** [3] *Also, the response concerning the choice of step-size is not convincing enough from a theoretical point of view... I also don't see how the noise is being treated here...* Again, we refer to the global response to all reviewers. We reiterate this: compared to the two time-scale updates in [59], the advantages of our approach are that (i) we *do not need* to explicitly reason about the relative step-sizes for the actor/critic updates. This makes the resulting algorithm more stable, while retaining the theoretical guarantees in Proposition 3. (ii) Since our analysis abstracts out how the critic loss is minimized (similar to [23,60]), we do not need to make assumptions about the noise. We hope that our response has clarified the reviewer's misunderstandings and better placed our algorithm in the context of existing analyses of actor-critic. We will explicitly include these comparisons and explanations in the final version of the paper. [a] Kumar et al, On the sample complexity of actor-critic method for reinforcement learning with function approximation, 2019. [b] Qiu et al, On the finite-time convergence of actor-critic algorithm, 2019.
Thank you for addressing my concerns. While your explanation on the relative stepsize helps clarify things, I still have some reservations. I understand your intention to isolate the effect of the critic loss, and the monotonic increase in $m_c$ and $m_a$ does make sense for enhanced performance. However, it would be greatly beneficial if you could provide additional support for the theoretical analysis in Proposition 3, e.g., by showcasing empirical evidence of your algorithm's superior performance compared to its counterparts in scenarios where there's insufficient critic update, such as with small $m_c$ or larger stochastic error, you could offer a more robust validation of your approach. My evaluation of the work still remain unchanged.
Experimental evaluation
Thank you for engaging with the rebuttal, and for your suggestion. There are three sources of error for an insufficient critic -- the bias (due to the limited capacity of the critic), the optimization error (due to small $m_c$) and the variance (the stochastic error due to insufficient samples). Our experiments demonstrate that when the bias dominates, using a decision-aware critic loss does indeed result in better performance. For the final version of the paper, we will include ablation studies varying $m_c$ and the number of samples, and hence compare the different methods when the optimization error or variance dominates.
The rebuttal addressed my concerns. I have adjusted my rating considering the rebuttal and other reviewers’ concerns.
Decision
Accept (poster)