Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning

>Benchmark Environment Selection Please refer to our Common Point 4 >The employed baselines use implementations that have not been optimized for the DeepMind Control suite The question is fair, BUT we have taken this issue into consideration in our implementations when reporting results. Please read our implementation details of the paper (Appendix C) and Common points 3 more carefully. Specifically, 1. We use default environment setting (1000 steps horizon), which is well accepted as shown in previous papers [1,2,3]. 2. The length of training at 1e6 maximum steps is also well accepted in the community [2,3]. 3. we use the same common hyperparameters for all algorithms. This choice of hyperparameters is well accepted as shown in previous papers [1,2]. As for the reference provided by the reviewer, it uses CUSTOM hyperparameters, while our selected set of hyperparameters are i) consistently used throughout our evaluations, and ii) also used in seminal works such as original TD3 and SAC. >Theorem unclear. the authors introduce what they refer to as the "step random estimation bias $\psi$", but do not use $\psi$ in any of the main statements of the Theorem. They then re-introduce the same quantity by copying the exact same description in Theorem 2 In Lemma 1, we show how TD error can be used to measure estimation error by formulating the step estimation bias $\psi$ and the function estimation bias $\Psi$ respectively as \begin{equation} \mathbb{E}[\Psi^{k+1} ] = \sum_{t=k+1}^{\infty} \gamma^{t-k-1} \mathbb{E}[\psi^t_{\theta'_\zeta}]. \end{equation} This equation highlights the significance of the step estimation bias $\psi$ in deriving our theorem which can use step bias to predict function estimation bias in an unbiased way. For a comprehensive understanding of how this bias contributes to our results, we suggest the reviewer read the discussion/proof presented in Common Point 2 and the full proof of Theorems in the Appendix, which includes a further elaboration during rebuttal. This part of our work underscores the integral role of Lemma 1 in supporting the theorems. Regarding Theorem 3, we build upon the foundation laid in Theorem 2 to make a statement about the impact of suboptimal actor updates on the critic. Specifically, we address how these updates influence the Q value (represented in the critic) in two distinct scenarios: overestimation and underestimation. \begin{equation} \mathbb{E}[Q_{\theta_1}(s_k,\pi_{DPG}(s_k)] \geq \mathbb{E}[Q_{\theta_1}(s_k,\pi_{TDR}(s_k))] \geq \mathbb{E}[Q^\pi(s_k,\pi_{True}(s_k))], \end{equation} and in the underestimation case, \begin{equation} \mathbb{E}[Q_{\theta_1}(s_k,\pi_{DPG}(s_k)] \leq \mathbb{E}[Q_{\theta_1}(s_k,\pi_{TDR}(s_k))] \leq \mathbb{E}[Q^\pi(s_k,\pi_{True}(s_k))]. \end{equation} These relationships illustrate how the quality of actor updates, whether suboptimal or not, can significantly affect the critic's performance in terms of Q value estimation. We believe this concept is clearly presented and does not lead to any confusion. >Novolty and LNSS issue Please refer to our Common Point 2 for novelty of TDR. LNSS is not among our claims for novelty in this study. Just like n-step to D4PG, LNSS is a useful add-on. If the reviewer is interested in LNSS, please refer to their paper [4]. As for the strategy of "lowering the pessimism to counteract the actor's Q-maximization," this approach is central to the functionality of TD3. In TD3, the method involves directly minimizing the Q value, which serves to mitigate the pessimism that arises as a byproduct of the actor's maximization of the Q value. However, as we've identified, this direct minimization approach can lead to other issues, as highlighted by the reviewer. Please refer to our Common Point 2, Theorem 1 (specifically case 1 and case 2) where by using TDR, we effectively addressed this issue by selecting a critic value that is associated with a lesser TD error. >"promotes a new problem of underestimation, which usually occurs during the early stage of learning, or when subjected to corrupted reward feedback or inaccurate states" (page 4). Yet this claim is not supported by any empirical or theoretical evidence in the text. Please refer to our Common Point 3. Consider at very early stage of training with $Q = 0$ as a common initialization. If the reward noise is negative, by inspecting the bellman equation, the estimation error will most likely result in an underestimation error. Similarly for inaccurate state and action due to $s_{noise}$ and $a_{noise}$, it is possible to have $\mathbb{E} [Q(s_{noise},a_{noise})] < \mathbb{E} [Q(s_{true},a_{true}]$ which will result in underestimation. This is why we need to study estimation error under noisy reward, state and action rather than a noise free environment. This also shows the significance and novelty of our work.

>The authors claimed that "The clipped double Q-trick is designed to solve the overestimation caused by the max operator." Actually, the max operator is not the culprit for the overestimation; we would encounter overestimation even using the bellman evaluation operator. The culprit is function approximation errors We would like to make the following two points. 1) The notion of overestimation error has long been established and frequently used in previous seminal works such as those we cited. For example, in Lan 2020 [2], published at ICLR, the first sentence of the second paragraph under section 3 says "The overestimation bias occurs since the target $max_{a' \in A} Q(s_{t+1},a')$ is used in the Q-learning update." TD3 [4] states in the introduction that "Overestimation bias is a property of Q-learning in which the maximization of a noisy value estimate induces a consistent overestimation". In Thrun \& schwartz 1993 [3], under section 2, it says that "The max operator (in Q learning), however, always picks the largest value, making it particularly sensitive to overestimations." 2) About the reviewer's comment that "we would encounter overestimation even using the bellman evaluation operator. The culprit is function approximation errors", please refer to our Common Point 3 where we discussed causes/sources of estimation error, which is NOT ONLY just the function approximation error. It can also come from noisy reward, state, and action. This is the very reason of our experiment design, and is considered one of our major contributions to studying estimation error at the root cause. >Unreliable Baseline Performance Incorrect assessment. Please refer to our Common Point 3. >Need More Experimental Results. Add more experimental results on MuJoCo and DMControl tasks. The reviewer may have missed reading SOTA developments and results on DRL. Please refer to our Common Points 3 and 4. Additionally, please note that Mujoco [1] is indeed a physical simulation engine. But the DeepMind Control Suite (DMC) has established its position in DRL as providing benchmark environments, which are built upon the Mujoco simulator. >SOTA Baseline and some related work First, on using SOTA baseline, please carefully review our Common Point 4. Additionally, our selection of off-policy baselines, as substantiated by extensive results based on the Deepmind Control Suite [5,6,7], represents the state-of-the-art (SOTA). Specifically, D4PG is widely recognized for its superior performance in continuous control problems, a fact underscored by its impressive scores, frequently surpassing 900 on a scale of 1000. This level of achievement has led to D4PG's extensive use in numerous significant and recent publications as a baseline method for comparison [8,9,10,11]. In regards to a comparison with BAC, please read our Common Point 3. Next, we noted that BAC is currently under review for ICLR 2024 and is yet to be officially published. Our criteria for choosing baselines prioritizes algorithms that are not only published but have also gained widespread acceptance in the research community. This approach ensures that our comparisons are made against benchmarks that are established and validated in the field, and the results can be repeated. One more important detail that we'd like to point out - BAC only considers adding a single noise to the action. This does not allow a sufficient study of the estimation error issue. In response to the recommendation to include RND in our comparison, it's important to note that RND primarily focuses on environments such as Montezuma’s Revenge, which is not a continuous control problem. As for REDQ, its evaluation predominantly relies on the OpenAI Gym benchmarks. According to benchmark studies such as Pardo 2020 [6], it has reached a consensus that methods such as SAC and PPO, which may perform well in Gym environments, show less impressive results in DMC benchmarks. As such, when consider using the more established DMC benchmarks. As previously mentioned, D4PG stands out as a well-recognized SOTA method within the community.

>Move Table 1 to the Appendix We thank the reviewer for the suggestion. But this table is highly informative, important, and comprehensive as it compliments the learning curves and other plots by providing quantitative performances upon learning convergence. This approach is standard and has been a common practice in the community. It can be seen in almost all seminal papers such as TD3 and SAC. >Better not to use the limited results to answer Q1-Q6 questions repeatedly We thank the reviewer for the feedback but we respectfully disagree. In quantitative analysis of experimental results employing both horizontal and vertical comparison analyses is necessary and essential. This is a standard technique in scientific data analysis on different performance measures. They answer different questions. Additionally, our presentation of using both horizontal and vertical comparisons actually helps guide readers to read the results and verify our claims. [1] Mujoco website https://mujoco.org/ [2] Lan, Qingfeng, et al. "Maxmin q-learning: Controlling the estimation bias of q-learning." arXiv preprint arXiv:2002.06487 (2020). [3] Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale,NJ. Lawrence Erlbaum,1993. [4] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018. [5] Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., ... \& Riedmiller, M. (2018). Deepmind control suite. arXiv preprint arXiv:1801.00690. [6] Pardo, F. (2020). Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. arXiv preprint arXiv:2011.07537. [7] Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Tb, D., \& Lillicrap, T. (2018). Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617. [8] Huang, S., Abdolmaleki, A., Vezzani, G., Brakel, P., Mankowitz, D. J., Neunert, M., \& Hadsell, R. (2022, January). A constrained multi-objective reinforcement learning framework. In Conference on Robot Learning (pp. 883-893). PMLR. [9] Gulcehre, C., Wang, Z., Novikov, A., Paine, T., Gómez, S., Zolna, K., \& de Freitas, N. (2020). Rl unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 7248-7259. [10] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., \& Bellemare, M. (2022). Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in Neural Information Processing Systems, 35, 28955-28971. [11] Chen, X., Mu, Y. M., Luo, P., Li, S., \& Chen, J. (2022, June). Flow-based recurrent belief state learning for pomdps. In International Conference on Machine Learning (pp. 3444-3468). PMLR.

>The comparative experimental results are not sufficiently reliable. None of the baselines are designed for addressing the estimation error problem. We thank the reviewer for reading our paper and for providing comments. However, we respectfully disagree with the reviewer on this point. Actually, our experiments were designed for the purpose of addressing estimation error. Please refer to our Common Points 2 and 3. Furthermore, please note that the issue of estimation error is of great importance regardless if the baseline algorithms are designed for this purpose or not. Refer to Common Point 3, estimation error poses a great challenge for almost all DRL algorithms. As demonstrated in our results (Figures 4, 5 and Table 6 in the updated paper pdf during rebuttal), if we deliberately introduce those factors that may cause estimation error (such as adding noises to $s, a, r$, all baseline algorithms show a significant performance drop from that without noise using the same benchmark environments. However, by using TDR, the lost performance were all compensated for to a large degree, and the baseline methods with TDR still produce stable results. This clearly shows our contribution and the novelty of TDR. >Will the parameters of the comparison algorithms in the experiment be consistent with the original paper (the best situation), and will their performance be worse than the original paper Please refer to our Common Point 3. If we evaluate under the Noise-Free conditions of the original papers, that defeats the purpose of this paper which aims at adequately addressing estimation errors that may come from multiple sources ($s, a, r$, and $\theta$). In real-world applications, these factors are prevalent and ubiquitous. Nonetheless, in the rebuttal, we have provided new data to further strengthen the novelty and significance of TDR. Please refer to Common Point 1 for our evaluations of baselines without Noise. >The paper emphasizes that the Twin TD-Regularized method can effectively control errors, but the effectiveness of bias control has not been demonstrated in the experiment section. We adhere to the same performance measures used in prior works such as TD3 and SAC, both of which aim to address estimation error. Please refer to our Figures 2,4,5, under both noisy and noise free case, TDR boosts average reward of baseline methods. We also remind the reviewer to read our Common Points 3 where we clearly explained why by adding multiple sources of noise, our baseline results no long replicate those published results (because they did not use noise). >The first Q-network is traditionally chosen to compute the actor TD regularization term. Will it have a negative impact on the selection of Q-network target values in the proposed method? Please refer to Common Point 2 where as we discussed that, following the TD3 convention, we use the first critic to compute actor. This is because theoretically $Q_{\theta_1}$ and $Q_{\theta_2}$ converge to the same result, making it redundant if we go between the two. For impact on the selection of Q-network value please refer to common point 2. >In subsection 3.4, the paper only analyzes the advantages of TDR in theory compared to the previous TD-Regularized actor network method. It should be demonstrated through some experiments. To highlight our points, We are now also providing additional results in the rebuttal Figure 7 of our updated paper. Our TDR actor leads to reduced learning variance, in agreement with the predictions made in Theorem 2. This improvement in our TDR actor's performance may primarily due to the discussion outlined in Common Point 2 that use both online critic can better measure the estimation error of critic updates. >In page 7, the meaning of "TDR has helped successfully address the random initialization challenge caused by random seeds" is confusing and difficult to understand, and it needs further explanation. Please refer to one of our cited references by Henderson et al. [1]. In their Figure 5, it is clear that initializing the environment, numpy, and other packages with varying random seeds can lead to results with significant variance in learning performance. Therefore, the ability of a DRL method to effectively overcome this challenge becomes a critical performance evaluation metric.

>The paper only demonstrates that TDR can improve the performance of baselines. You should compare some SOTA bias control algorithms in recent years. It's important to note that TD3 and SAC, the two baselines we've compared TDR against, are not only well-accepted in the field but also have demonstrated efficacy in directly addressing overestimation error, a key aspect of bias control. Additionally these two methods have been used as SOTA baselines in most recent top conference papers [4-7]. Our evaluations focus on showing the effectiveness of TDR within the context of general off-policy methods. By applying TDR to these widely recognized and established baseline methods (TD3, SAC, D4PG), we aim to demonstrate its capability to enhance performance in more realistic (aka noisy) environments. >The ablation studies are not quite thorough. You should provide more experimental results about approximate estimation error for clearly suggesting the benefits of different components of the algorithm. The paper proposes to combine TDR with LNSS method effectively, but LNSS is not the work of this paper and is not suitable for ablation experiments. We respectfully disagree with the reviewer and here is why. In our evaluations, we follow the same established protocol in widely accepted and sound research works. We recommend the reviewer read TD3 and D4PG more carefully to find out. TD3's core innovation lies in its Clipped Double $Q$ Network, accompanied by additional features such as Delayed Policy Update and Target Policy Smoothing Regularization. In their ablation study, the authors of TD3 assessed the impact of each individual component by removing it from the TD3 framework. Likewise, D4PG's primary advancement is the integration of distributional learning into DDPG, enhanced by additional elements like Prioritized Experience Replay (PER) and n-step updates. The effectiveness of each component in D4PG was evaluated by incrementally adding them to DDPG. In a similar vein, our approach with TDR involves components like TD regularized critic (TD critic), TD regularized actor (TD actor), and an additional enhancement, namely LNSS. We analyze the contribution of each of these elements by incorporating them into baseline algorithms, thereby assessing their individual impact on performance. This approach allows us to understand the distinct value added by each component in the TDR framework. In terms of ..."You should provide more experimental results about approximate estimation error...", in addition to our evaluations provided in the paper, please also review our newly added data (Figures 4,5 and Table 6) that summarize TDR evaluations without noise. That actually is an evalution (and performance comparison) for approximation errors. [1] Henderson, Peter, et al. "Deep reinforcement learning that matters." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018. [2] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018. [3] Haarnoja, Tuomas, et al. "Soft actor-critic algorithms and applications." arXiv preprint arXiv:1812.05905 (2018) [4] Agarwal, Rishabh, et al. "Reincarnating reinforcement learning: Reusing prior computation to accelerate progress." Advances in Neural Information Processing Systems 35 (2022): 28955-28971. [5]Eysenbach, Benjamin, et al. "Contrastive learning as goal-conditioned reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 35603-35620. [6]Li, Chengshu, et al. "Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation." Conference on Robot Learning. PMLR, 2023. [7]Liu, Zuxin, et al. "Constrained variational policy optimization for safe reinforcement learning." International Conference on Machine Learning. PMLR, 2022.

>The main weaknesses of the paper are the novelty of the approach We thank the reviewer for reading our paper and for providing commons. However, we respectfully disagree with the reviewer. The TDR method is NOT an A+B+C work. Please refer to our Common Point 2, especially the one on Novelty of TDR. As a side note, LNSS is just a useful add-on, playing a role similar to PER in D4PG [3]. We never claimed any credit for LNSS as a (key) contribution of this work. >Experiment Results Reliable that the replicated baselines seem worse than original papers. We thank the reviewer for noticing this. Please refer to Common Point 3. >Section 3.1 is a review and may confuse reader Yes, and No. Section 3.1 is not only a background but also an introduction of necessary notations and problem formulation. This is very common, and necessary, in RL papers. Please refer to TD3, D4PG, SAC [1,2,3]. >How this sample based TD error serve as a sound measure for reducing the estimation error? Doesn't this lead to a large variance? This sample based approximation is a common approach in DRL. DPG [6] uses sampled TD error to approximate true gradient. DQN, DDPG, SAC, D4PG [2,3,4,5] use collected samples to approximate $Q$ values. TD3 [1] uses the Q value samples from two target $Q$ networks to address overestimation bias. In a similar vein, our TDR method uses TD error samples from two target networks to approximate both overestimation and underestimation errors. Regarding the variance, "the variance can grow rapidly with each update if the (estimation) error from each update is not tamed." as state by TD3 authors (refer to TD3 [1] section 5). Based on Theorem 1 and discussions of Common Point 2, we show that by using TDR, RL agents will result in better target values with less estimation error compared to previous clipped double $Q$ network in TD3. Therefore, following the same analysis as TD3, TDR will have the same or even more variance reduction. >Our TD-regularized actor network directly penalizes the actor’s learning objective whenever there is a critic estimation error.This statement is weird. It would be better to state this as "whenever the critic estimation error is non-zero". In addition, it is unavoidable to have estimation errors in practice when learning these objectives. We thank the reviewer for rephrasing the sentence for us. Actually there is no difference between how we and the reviewer said it. Nonetheless, we can update the phrase if the paper gets accepted. [1] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018. [2] Haarnoja, Tuomas, et al. "Soft actor-critic algorithms and applications." arXiv preprint arXiv:1812.05905 (2018) [3] Barth-Maron, Gabriel, et al. "Distributed distributional deterministic policy gradients." arXiv preprint arXiv:1804.08617 (2018). [4] Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013). [5] Lillicrap, Timothy P., et al. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015). [6] Silver, David, et al. "Deterministic policy gradient algorithms." International conference on machine learning. Pmlr, 2014.