Many reinforcement learning approaches rely on temporal-difference (TD) learning to learn a critic. However, TD-learning updates can be high variance. Here, we introduce a model-based RL framework, Taylor TD, which reduces this variance in continuous state-action settings. Taylor TD uses a first-order Taylor series expansion of TD updates. This expansion allows Taylor TD to analytically integrate over stochasticity in the action-choice, and some stochasticity in the state distribution for the initial state and action of each TD update. We include theoretical and empirical evidence that Taylor TD updates are indeed lower variance than standard TD updates. Additionally, we show Taylor TD has the same stable learning guarantees as standard TD-learning with linear function approximation under a reasonable assumption. Next, we combine Taylor TD with the TD3 algorithm, forming TaTD3. We show TaTD3 performs as well, if not better, than several state-of-the art model-free and model-based baseline algorithms on a set of standard benchmark tasks.
Paper
Similar papers
Peer review
Summary
This paper proposes an algorithm for improving model-based actor-critic methods in continuous state and action spaces. When using Dyna-style updates, the algorithm can compute an expected update over a small noise distribution using a linearization to reduce the variance of the performed update. This method is shown to be theoretically-sound in a simplified setting and a variety of experiments demonstrate the effectiveness of the proposed approach on standard continuous-control benchmark tasks.
Strengths
- The main idea is interesting, computing an explicit expected update over a small region by using a linearization. - The experiments are done well with meaningful baseline algorithms. In particular, the supporting experiments other than standard learning curves are a welcome addition: The variance analysis comparing sampled updates to the proposed one and the ablation study on cosine similarity vs. the inner product (in the appendix). - The theoretical result in the linear setting is nice even if it is in a simplified setting and the proofs are correct as far as I can tell.
Weaknesses
There are some aspects of the presentation that I think could be improved. These are more minor points overall though. - For example, I would avoid using the term "Monte-Carlo" estimates when discussing sampling of TD updates since it can lead to some confusion due to "Monte-Carlo" often being used to refer to MC estimates of the return in contrast to the bootstrap estimates that TD uses. I would consider using the phrase "sampled TD update" instead of "MC TD update". E.g. line 31, line 42, ... - I would also suggest including a line about continuous state and actions in the introduction to clarify the problem setting since the proposed algorithm would mainly be applicable there. - Section 3.1 describing the updates was a bit difficult to follow at first. Some of the notation such as using $\Delta_{Exp}$ was slightly confusing since it suggests that it is the overall expected update even though it's only the expected update over $\xi_a$. An alternative could be $\Delta_{\mu_a}$. The expectation $E_{s,a$}[\Delta \theta(s,a)]$ is slightly unclear as to what distribution is used for $(s,a)$ here.
Questions
- Why does aligning the gradients w.r.t the action of the Q-value and the TD error make sense? Is there an interpretation for this? It seems like the inner product between those gradients could be a meaningful quantity more broadly e.g. as an evaluation metric for the quality of a critic. - What is $\eta$ in equation (11)? It wasn't described in the text or the appendix. - This method could potentially also be used in cases when the simulator is differentiable to start with. Have you experimented with this at all? - Minor point: I would consider modifying the title of the paper to a more descriptive name which could reference model-based RL or Dyna-style updates.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations are discussed.
Summary
This paper, 1. Proposes Taylor TD, a model-based RL algorithm that uses a Taylor series expansion to analytically estimate the expected TD update over a distribution of nearby state-action pairs. This reduces variance compared to standard MC TD updates. 2. Provides theoretical analysis showing the variance of Taylor TD updates is lower than standard TD updates. Also proves stability of Taylor TD with linear function approximation. 3. Empirically demonstrates lower variance of Taylor TD on several RL tasks. 4. Combines Taylor TD with TD3 in an algorithm called TaTD3. Shows strong performance of TaTD3 relative to model-free and model-based baselines on MuJoCo benchmarks.
Strengths
1. The core idea of using Taylor expansions to estimate expected TD updates is novel and well-motivated from a theoretical perspective. This analytic approach to reducing variance is interesting. 2. The method preserves the convergence guarantees of TD-learning under linear function approximation, as shown formally. This is an important theoretical contribution. 3. The variance reduction analysis provides evidence Taylor TD reduces variance versus standard TD methods. 4. The algorithm is straightforward to implement on top of existing model-based RL frameworks.
Weaknesses
1. In Figure 2, Taylor TD3 seems to provide only very small performance improvements over the baselines in most environments (4 out of 6). The gains are noticeable in only 2 environments. Given that Taylor TD3 is much more complex, is the minimal improvement in the majority of environments concerning? The results seem to imply the benefits may be limited to certain environments. Some discussion of why the gains are so marginal in certain tasks would be useful. 2. Some derivations are not clear. It is not clear what loss function are the authors using to learn the model.
Questions
1. In the Taylor TD update derivations (Equations 12, 17 to 18), the Hessian terms present after the Taylor expansions disappear in the final update equation. Can you clarify the assumptions or steps that lead to the Hessian terms dropping out? As these terms arise directly from the Taylor approximations but then vanish, an explicit explanation should be provided about how and why they drop out of the final update form. 2. It is also not clear how the loss function (equation 18) is derived. Could you please provide the derivation? Algorithm 1 includes an update equation for the model parameters w using some loss L, but does not define what this loss function L is (unless I am mistaken). Please specify what objective or loss function is used to optimize the model parameters w based on the observed environment transitions. 3. The current evaluations of Taylor TD are limited to a small set of MuJoCo continuous control tasks. Did the authors consider testing on additional environments that evaluate aspects like partial observability, sparse rewards, and higher task dimensionality? A more diverse test suite could provide greater insight into the strengths and weaknesses of Taylor TD across different conditions. What other test beds do you think could be useful for thorough analysis? What are the challenges that you foresee? 4. The hyperparameters indicate high values for \lambda_a and very low values for \lambda_s. Why is that the case?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The authors mentioned about the limitations in the last section and I do not forsee any negative societal impacts.
Summary
The author presents a variance reduction trick in TD learning by taking the analytical expectation of the gradient using first order taylor expansion. Standard RL, replaces the expectation over the gradient of critic (Q-value) with a sampled value from the replay buffer or online learning samples. The paper proposed to replace this expected gradient of Q value function with first order taylor series expansion of the same. They proposed taylor expansion over both state and action. ** I have read the rebuttal of the authors.
Strengths
Originality 1. The work is proposing novel research in the direction of stabilizing the critic updates by reducing the variance using first order taylor approximation of the expected gradients. Quality 1. The paper could be improved on how the experiments are conducted for Fig 1. Furthermore provide better clarity on the questions section later. Significance 1. The work is definitely targeting a significant problem that could have impact on RL algorithms in general. This work is stabilizing the critic updates by reducing the variance.
Weaknesses
The weakness/questions are mentioned in the next section. If the authors provide justification for the below, I am willing to change the score. Add references for the other variance reduction work in direction of - variance reduction of policy updates, inherent stochasticity (risk-sensitive RL). They are very related to your proposed research.
Questions
1. It is not clear in the paper how the model (transition, reward) is learnt using the $L_\theta$ objective in Eq 18? Algo 1 shows the update $w$ (model parameter ) with the gradient of $L_\theta$ without showing how $w$ influences it. 2. Why is $\mu_a$ in Eq 19 sampled from deterministic target policy? Why do you enforce this determinism on the policy being learnt? 3. In Fig 1, what is the variance computer over? Is it same $(s,a)$ pair and then compute multiple gradients over that? Is it batch data, comprising of same data for both MC TD and Taylor-TD and then the variance over updates is computed? Because if the variance is computed over different state-action, then the high variance could be a factor of that (s,a) pair having high stochasticity in the dynamics. It wouldn’t mean necessarily that updates have less variance because of taylor approximation. Please explain what this variance update is over. 4. What is the effect of reducing the variance in critic update vs to that actor update(Ref[1])? Have you tried ablation study for the same - which one is better, or what effects they have on learning separately vs together? 5. Provide some connections with Risk-sensitive RL and robust RL - inherent stochasticity vs imperfect knowledge? Will be good to connect to the literature in that area (add references for same)! 6. Comparison with expected sarsa kind of update in tabular environments? That would help to understand how expected update over the gradients help over Expected sarsa. (Would be good to know more insights on this, just something additional) References -- [1] Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
Yes the authors have included the limitation section. The negative societal impact section doesn't apply to this line of research.
Summary
The authors are proposing a method for reducing the variance of TD-learning updates for model-based RL applied to problems with continual state-action spaces. The method relies on Taylor expansion of noise terms in action and initial state distribution, reducing the contribution of those to the variance of TD-update. The authors demonstrate variance reduction both theoretically and empirically, as well as performance on par with SOTA model-based methods such as MBPO (Janner et al, 2019) and MAGE (D'Oro et al, 2020). For empirical evaluation, the authors apply the proposed update adjustment to TD3 with Dyna.
Strengths
The empirical results seem pretty strong, though I would have found results with longer training times to be more convincing (about 2x longer, i.e. as long as was used in other work such as Janner et al, 2019). I appreciated the strong theoretical backing, i.e. demonstrating lower variance and stability guarantees (though it should be noted that I did not check the proofs as this work falls outside of my expertise).
Weaknesses
Minor: - the authors should be clearer in the main body of the paper about the additional computational demands of the proposed method - some of the particularly interesting results can be only found in the Appendix. For example, the claim that the method performs better on large state-action spaces seems like an important one, and it was good to see it explored in a more controlled setting in Appendix F. Similarly, we can only find ablations (importance of state expansion and cosine similarity) in Appendix I (also it would be good to see these results on all 6 environments). I hope the authors can find a way to move these results to the main body of the paper (personally I found those to be more insightful than the comparison in Fig 3).
Questions
1. In Figure 1, does the variance of each method change during training? 2. In Figure 2, how do the results look like if the models are trained for another 2x steps? What about comparison in terms of training time?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
No concerns regarding potential negative societal impact, and the limitations were discussed to a reasonable amount.
Thank you
Dear Authors, I appreciate the thorough responses to all my questions. My concerns on derivation, loss function and others have been addressed. I adjusted the score accordingly. Thanks
Thanks for the responses
Thanks for addressing some of my concerns. I had some additional input. Please clarify more on the third point mentioned below. 1. "These methods tackle the variance at the level of the policy rather than critic updates by also integrating over the stochasticity induced by the action distribution." -> There exists work in risk-sensitive RL literature that tackles variance estimation by directly using the Bellman operator for the variance. Would be good to include them in references too. Also it would be good to provide clarity on how it differs from these works. I agree that your work minimizes the variance of the value function estimation by using Taylor approximation. [1]Sobel, M. J. 1982. The variance of discounted Markov decision processes. Journal of Applied Probability 19(4): 794– 802. [2]Jain, Arushi, et al. "Variance penalized on-policy and off-policy actor-critic." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021. [3]Tamar, A.; Di Castro, D.; and Mannor, S. 2013. Temporal difference methods for the variance of the reward to go. In International Conference on Machine Learning, 495–503. 2. “It is not clear in the paper how the model (transition, reward) is learnt” -> If the transition and reward are learnt using maximum likelihood, please change the notation in Algo 1, the model update step where the same \mathcal{L} was used. The same \mathcal{L} is used in Eq 18 which makes it very confusing. Further, also describe the notation in intro/Taylor TD section - how the model was learnt. 3. Why is the standard TD method called (MC) TD? Because in the TD method, we are still estimating the target by bootstrapping with a 1-step Q value. I can't understand why MC is there. In this work, the “standard TD style” is also used. Could you clarify the difference between the two?
Response
Thanks for your continued engagement! [1,2,3] look like great papers. In fact, they're super-relevant for some of our other work, so I will pass them on. And of course, we will include them in the related work for this paper. But importantly they are doing something quite different from this paper. [1,2,3] are all considering variance _in the return_, induced either by stochastic _rewards_ or _transitions_. In contrast, this paper is not considering variance in the return, nor does it considering variance that arises through stochastic rewards or transitions. Instead, we are considering variance in the _TD update_ induced by stochasticity in the _choice of initial action_ (and visited states). [1] Sobel, M. J. 1982. The variance of discounted Markov decision processes. Journal of Applied Probability 19(4): 794– 802. This paper presents formulas "for the variance and higher moments of the present value of single-stage rewards in a finite Markov decision process". [2] Jain, Arushi, et al. "Variance penalized on-policy and off-policy actor-critic." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021. This paper proposes "on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return." [3] Tamar, A.; Di Castro, D.; and Mannor, S. 2013. Temporal difference methods for the variance of the reward to go. In International Conference on Machine Learning, 495–503. (The title is a reasonable summary.) > If the transition and reward are learnt using maximum likelihood, please change the notation in Algo 1, the model update step where the same \mathcal{L} was used. The same \mathcal{L} is used in Eq 18 which makes it very confusing. Further, also describe the notation in intro/Taylor TD section - how the model was learnt. Good catch. We'll definitely update this as part of the more general revisions regarding the model learning. In particular, we will use $\mathcal{L}^{\text{model}}$ as the model loss, and $\mathcal{L}^{\text{critic}}$ as the critic loss (e.g. in Eq. 18). > Why is the standard TD method called (MC) TD? Because in the TD method, we are still estimating the target by bootstrapping with a 1-step Q value. I can't understand why MC is there. In this work, the “standard TD style” is also used. Could you clarify the difference between the two? We're using Monte-Carlo (MC) as opposed to Taylor (Ta) to emphasise that there are two ways to compute the expectation in Eq. 7. We could draw many samples of $\xi_i$ and compute an empirical average. That's a Monte Carlo approach. Alternatively, we could use our proposed approach of computing the analytic expectation for a first order Taylor series expansion. We realise that there is a bit of a clash with an alternative use of "Monte-Carlo" in RL (specifically Monte-Carlo estimates of the return). Following Reviewer 9ew5's suggestion we will switch to using "sampled TD update" rather than "MC TD" to avoid this potential source of confusion.
Reviewer response
Thank you for your clarifications regarding the notations and experiments. I've revised my score now. Your paper presents a substantial technical contribution to the field. However, refining the writing will greatly enhance its accessibility to a wider audience. As a fellow reviewer, I also needed some time to fully understand the core contributions. I strongly encourage the authors to focus on improving the readability, as this will undoubtedly solidify the paper's strength. Your work is really good, and with some writing adjustments, it can become amazing.
Completing paper discussion
Dear Reviewers, Thank you for your efforts here. If you haven't already, please acknowledge the authors responses. Also please read other reviews and let the authors know if additional concerns have been raised. Reviewer 9c2H: Thank you for engaging in discussion to see if the authors can clarify your concerns! Thanks, Your AC
Response
Thank you for the response and consideration. About the presentation: > Alternatively, we could further clarify the definition of in Eq. 7? Let us know if you have a preferred option. Perhaps clarifying the definition would be sufficient and re-iterating that $\mu_a$ is fixed and the only randomness is over $\Xi_a$. I agree there's some difficulty in choosing the right notation here and leave it to your discretion. >Something like "Taylor TD-learning in model-based reinforcement learning"? We'd be happy to make that change. That sounds good. I appreciate the explanation of the alignment of TD-error and $Q$ value gradients. It makes sense to me although it doesn't necesssarily explain why this direction would be useful for policy optimization. There may be some deeper reason here to be found. I am satisfied with the response and the references to risk-sensitive RL by Reviewer 9c2H would be a nice addition too. I would also encourage some investigation regarding the alignment of TD-error and $Q$ values to be done even if it is preliminary. Overall, I will still recommend acceptance.
More details about runtimes
We have got some numbers around the runtimes for our method (TaTD3) against the competitive baselines (MBPO and MAGE). MAGE has about the same runtime as our method (TaTD3), while MBPO is a lot slower. | Method | Pendulum | Walker | Ant | Humanoid | | --- | ---- | ---- | --- | --- | | MAGE | 36 s | 63 s | 75 s | 127 s | | MBPO | 52 s | 133 s | 158 s | 235 s | | TaTD3 | 38 s | 68 s | 72 s | 117 s |
Decision
Accept (poster)