Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control

Summary

This paper addresses the problem of exploration in ensemble-based reinforcement learning. It introduces a novel approach called Trajectory Aware Ensemble Exploration (TEEN) that aims to enhance exploration by increasing diversity among the sub-policies. This diversity is measured by the KL divergence between the distributions of the trajectories. The method achieves diversity by optimizing the variational lower bound, which is then incorporated into the policy gradient step with the addition of a regularizer term. To reduce estimation bias on the Q function, the method selects the minimum value from a random set of critics based on the average Q value of the actions chosen by all sub-policies. The paper provides theoretical analysis to support this design choice, demonstrating that it reduces both estimation bias and variance. The effectiveness of TEEN is validated through experimental results on MuJoco and Deepmind Control Suite. These results show that the proposed method successfully explores a wide range of states and surpasses the performance of existing DRL algorithms in continuous control, including those specifically designed for efficient exploration.

Strengths

- It proposes a new method that improves the exploration in ensemble-based RL by increasing the diversity of the sub-polices in terms of their state-action visit distribution. - The implementation is straightforward and involves two steps on top of a DDPG-like method: - add a regularizer to the policy gradient - add a routine to select and update sub-policies and to compute the target Q value - It provides theoretical analysis to justify the choice of target Q value - Experimental results are presented to validate the efficacy of the method.

Weaknesses

- There appears to be an inconsistency between Line 18 in Algorithm 1 and the text. Specifically, Line 18 suggests that all sub-policies are updated sequentially, while the "recurrent optimization" section indicates that only one sub-policy is updated in each gradient step. Could you clarify this discrepancy? - Theorem 1 demonstrates that the method reduces the expected Q value. However, the experimental results indicate that the algorithm tends to underestimate the Q value. It's unclear whether underestimation is preferable to overestimation. If possible, it would be beneficial to address this aspect in the analysis. - The main paper frequently refers to Appendix C, but it appears to be missing from the appendix section. - In Line 436 of the appendix, it seems that the last expression is missing a multiplier p(s). Could you confirm? - There's a mismatch between Lemmas 2 and 3 in the appendix and the content presented in the main paper. - The proof for (v) in Lemma 3 does not show the variance inequality: V[X_{1:{N+1}}] \geq V[X_{1:{N}}]. - Formatting Issue: - The readability could be improved if Table 2 used the same row/column heading as those in Table 1. - Appendix line 483: Table index is missing. - Grammar: - Line 151: we can “increase” this equivalent optimization target. “maximize” seems more appreciate here. - Line 272: “.While purely ensemble multiple models may not certainly improve the performance.“ - Line 273: “it does not take effect“, “evenly degrading”.

Questions

Please refer to the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

- The authors have discussed one limitation regarding the sensitivity about reward scaling in Sec. 7. - Could the authors comment on the computational overhead of the ensemble?

Summary

This paper presents a new ensemble RL algorithm. The main motivation of this work is to increase the variation between sub-policies, which is measured by the distance between the corresponding state-action visitation distribution. It formulates the loss function by connecting the visitation distribution distance to the mutual information theory and introduces a discriminator function to measure the policy variation. It also proposes a few tricks to improve the exploration effectiveness of the proposed formulation. The method is experimented on a few Mujoco and Deepmind control suite tasks and compared against several off-policy RL methods and one ensemble RL method.

Strengths

1. The proposed method is motivated by an intuitive idea that increasing the trajectory variation of the sub-policies may improve the exploration and performance of the ensemble policy. 2. The results show the proposed method consistently outperforms baselines across a wide span of tasks in different benchmarks. 3. The experimental analysis is helpful to understand the secret sauce behind the proposed method and the effect of different hyper-parameters.

Weaknesses

1. The exposition of the paper need improvement (more in the next section) 2. The evaluated control problems are relatively simple and the approach needs to be evaluated on more complex tasks.

Questions

1. Is the statement for the overestimation problem in DDPG properly demonstrated? Eq(4) (line 113) seems to be the formulation for discrete action space while this paper and DDPG focus on continuous action space. Please refer to [[9]](https://arxiv.org/pdf/1802.09477.pdf) for details. 2. It is unclear what the latent variable $z_k$ is at the beginning when it is first presented. Could be more clear if it just says $z_k$ is a categorical index or one-hot encoding of the sub-policy selection. 3. Line 146 mentions using KNN to measure Eq (8) but does the proposed approach indeed use it? If so, where is it applied? 4. Eq (9) is an important step to connect state-action visitation distribution distance to mutual information. Any references or derivation for Eq (9)? 5. The Recurrent Optimization trick (Line 169) is adopted to prevent sub-policy from exploration degradation. If I understand correctly, it corresponds to Line 18-19 in Algorithm 1 where only the sampled policy $\pi_k$ is updated. However, the gradient of the loss function (Eq (14)) should be independent among sub-policies. Could the authors explain how will the sub-policy $\pi_{\phi_k}$ change if we update all sub-policies instead of just $\pi_{\phi_k}$ in Algorithm 1 (Line 18-19). 6. **Algorithm 1** helps understand the method while there is no reference to Algorithm 1 throughout the paper. 7. How is the discriminator function $q$ trained? I could not find any details about it. 8. How does the presented algorithm perform on more complex control tasks? The current evaluated tasks are relatively simple that do not require a sophisticated amount of exploration of the policy. It would be more convincing if the approach can be benchmarked on Humanoid and Dextrous hand manipulation tasks. 9. The appendix is incomplete.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations have been discussed above (i.e. it is unknown about the performance of the presented approach on complex tasks).

Summary

This paper presents an algorithm called trajectories-aware ensemble exploration (TEEN) for sample efficient ensemble reinforcement learning. The authors point out that the existing ensemble methods do not effectively address the required diversity in exploration, which TEEN is designed to tackle better. It achieves this by encouraging diverse behaviors through exploration of the state-action visit distribution measure space. Theoretical results are also presented to show how the design principles of TEEN could encourage diversity in exploration, while the experiment results demonstrate its empirical success in popular benchmark tasks.

Strengths

This manuscript presents an interesting thesis that sample efficient ensemble reinforcement learning methods require diverse exploration in sub-policies. The authors discuss this topic in detail and systematically approach the proposed solution. I view the finding of the diversity requirement of sub-policies coupled with the provided theoretical analysis and the limited yet promising experimental results as the main strength of the work. I, therefore, believe if properly executed, this can be a valuable contribution to the community.

Weaknesses

One of the main limitations of the work is its limited evaluations. The authors evaluate the proposed TEEN method to answer three main research questions. But I find their evaluations lacking the experimental rigor that is needed to fully appreciate the proposed method. First, in RQ1 in evaluating the performance against baselines, the proposed method shows visible benefits in Mujoco tasks, which is promising. But my concerns are that all four tasks evaluated may contain similar exploration challenges. Given exploration is one of the key focuses of the work, I would like to see the performance benefits across different tasks that pose different exploration challenges. Perhaps this is what the authors are attempting to do with DeepMind control experiments, but despite the mention, I did not find such experiments in the paper or in the appendix (Appendix C is empty). Regarding RQ2 and RQ3, my concerns are with the fact that it is only based on a single task (Ant-v3). For both questions, it is not significant enough to make claims based on a single task result. Furthermore, for RQ3, while I never found the results on the DeepMind control suite, the authors mention the best $\alpha$ is 0.02. This is a very low $\alpha$ and almost as having no trajectory-aware exploration. This further questions my concerns about how generalizable the proposed method is across different tasks. Clarification is appreciated. Overall for experiments, authors use only TD3 as the learning algorithm. I would like to see if the observed performance gains are dependent on the choice of the learning algorithm. It is also fine if there is an observable reliance, but in that case, I would like to see the contributions rephrased accordingly. I presume training the proposed model is challenging, specifically when it comes to finding the best ensemble size N and number of target values M (due to the trial and error approach in finding them, as in RQ2). It also appears that this training would require considerable computations. I would like to see a discussion of these limitations (and other limitations) of the method. If the stated limitations are addressed, particularly related to the evaluation process, the proposed work can be of benefit to the community. Consequently, I would be inclined to reconsider my score if the authors can effectively rectify these issues.

Questions

1. Shouldn’t the equation 11 be $\log(N) - \mathbb{E}_{s,a,z}[-\log(\rho(z|s,a))]$? 2. Also, shouldn’t the LHS and RHS of equation 11 be approximately equal and not equal? 3. Define $\mu$ and $\sigma$ in theorem 1. 4. While I did not check the proofs carefully, it appears the authors make simplification assumptions in the proof of theorem 1 that is not stated in the theorem (ex: zero-mean distribution). Please state them properly in the theorem.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please discuss the limitations of the work, as pointed out earlier. This will help readers better appreciate the work.

Summary

The paper presents an efficient ensemble learning strategy termed Trajectories-awarE Ensemble exploratioN (TEEN), which promotes exploration by diversifying the state-action visitation distribution of multiple sub-policies. The proposed method introduces a discrepancy measure that tries to maximize the difference in entropy of estimating the state-action visitation with and without the information of the policy (represented by the latent variable) and vice versa. Such a discrepancy is approximated by learning a discriminator that tries to differentiate between state-action pairs visited by different policies. And the log-likelihood of the discriminator is used as a regularizer for policy improvement along with maximizing the expected return of the policy. The authors employ additional strategies of updating a single policy at a time and clipping the discriminator log probabilities to ensure proper exploration. Finally, the estimation bias is controlled by calculating the target value as the minimum of the mean of the ensemble Q-values for each sub-policy. The choices are theoretically motivated and the algorithm is analyzed in simulation on Mujoco Control and DM Control Suite benchmarks.

Strengths

1. TEEN provides new insights into using the measure of the state-action visitation distribution to promote exploration along with maximizing returns in policy optimization. 2. Each design choice made is properly justified: namely updating one sub-policy at a time and clipping the regularizer gradients.

Weaknesses

1. There might be states not captured by the state-action visitation of one particular sub-policy (say $\pi_a$). In that case, the contribution of $\rho^{\pi_a}(s,a)$ to $\rho$(s,a) in eq (6) can be highly noisy and misinformative (garbage probability). Giving equal weightage to state-action visitation of every policy while calculating for the ensemble might be misleading. 2. It seems that the latent variable plays a key role in defining the algorithm and the discriminator. However, there is no discussion about the choice of $z_k$ and training of the discriminator in the paper or in Alg 1. How do you choose $z_k$ and how does the training take place of the discriminator? If done parallelly with the policies, the framework follows a min-max update rule, shown to be unstable in prior works like GAIL.

Questions

1. How do you decide the actions while inference? 2. Can you clarify more about the weakness pt.2? 3. Section C is missing in the Supplementary. 4. I am unsure, but it seems that TEEN uses very high random exploration steps as mentioned in the supplementary. Do you use same for all the other algorithms? 5. If the regularization is already clipped, why is there a coefficient ($\alpha$) required? Can a comparison be shown between these two cases? Also, $\alpha=0$ means that there is only a change in Q-value computations as compared to TD3, hence there must be a result for all the environments with $\alpha=0$ and $\alpha=0.2$ to show a more convincing contribution of the regularization term. Currently, it is only shown for one env (Ant) and it is particularly not clear how much the regularization term is contributing to the exploration in addition to the ensemble of sub-policies. 6. The timesteps shown in the results are timesteps for individual sub-policy or the whole ensemble i.e. (1M/10 = 100k steps for single sub-policy)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have mentioned the limitation as the tuning of $\alpha$.

Thanks for the detailed response

The rebuttal is responsive on all major points, and the new experiments further confirm the findings (performance and ablation results) and highlight the limitations (computational cost). Therefore, I will raise my score accordingly.

Further discussion would be welcome.

Dear Reviewer C6dk Thank you for your detailed review of our research. Your constructive comments are very helpful in helping us to improve our submissions. We truly value your feedback and have addressed your concerns in detail in our response. Have all your questions been answered? Since we have **less than** three days left for discussion, please let us know if there's anything else you'd like to discuss. If you're satisfied with our responses, we humbly hope you might consider giving the submission a **higher score** based on our response and your high comments on the Sound, Presentation, and Contribution. Best wishes, Authors.