Latent Exploration for Reinforcement Learning

Summary

This paper explores the exploration problem in RL algorithms by considering the action-space correlation. The approach is motivated by the fact that over-actuated systems with high-dof action space require correlated control actions. Such action-space correlation is ignored by action-space random perturbation in a standard RL algorithm which results in inefficient exploration. This work proposes to add random perturbation in the latent space in the last layer of the policy network to impose the proper action structure in the perturbation distribution. The presented approach can be potentially integrated into most existing RL algorithms to enhance the exploration capability and is shown to perform great, especially in high-dim musculoskeletal environments.

Strengths

1. The motivation of the work is clear and intuitive. The provided toy example is a very illustrative example to motivate the idea of using correlated action perturbation. 2. The proposed method is concise and seems to be universal in that it can be easily integrated into most RL frameworks to improve exploration efficiency. 3. The experimental results are convincing to show the effectiveness of the proposed Lattice method.

Weaknesses

1. The selection of the initial logstd for the $S$ matrices seems to be not straightforward and might need extra experiments to determine. 2. The presented Lattice method relies on a good learned policy to impose the action correlation structure, which might result in local-optimal policy (also see point 3 in the next section). 3. The exposition of the paper can be further improved to be more clear and consistent. a) The second motivating example (the one in simulation) is confusing. b) In Line 148, $N_l$ is used for the latent space size, while in Figure 1, $N_x$ is used for latent space size. c) Line 168: $\epsilon_a = P_a a \Rightarrow \epsilon_a = P_a x$ d) It was never mentioned in the main paper about the separate action space perturbation until $\Sigma_a$ appears in Line 152.

Questions

1. The two images in Figure 2(B) is hard to parse. Are those images obtained from the action perturbation training or the latent perturbation approach? 2. In the MyoSuite environments (Figure 4), I noticed Lattice-PPO (T=1) consistently works better than Lattice-PPO (T=4) except for the **Pen** task. Does it mean time-space correlation is not important in general? Do you have any idea on why T=1 works better than T=4 and what is special in **Pen** task that makes T=4 preferable? 3. The proposed Lattice approach benefits the action structure learned by the policy and imposes this learned structure to the action perturbation. However, on the other side is this also a limitation of this approach? Will Lattice suppress the emergence of novel control modes and result in a local-optimal policy? If this is not a problem, which part of the algorithm is the key to addressing this issue? 4. It would be valuable to also ablate the $P_ax$ term in the method. Is this the key to addressing the last point? 5. gSDE (T=4) seems to work consistently worse than other approaches (even worse than vanilla PPO) in all MyoSuite environments. It would be interesting to see the performance of gSDE-PPO (T=1) which differs from PPO on whether having state-dependent perturbation. If gSDE-PPO (T=1) does not work well either, then it might not be reasonable to have $\alpha$ to toggle between gSDE and Lattice. Instead, using $\alpha$ to toggle between regular RL perturbation and latent space perturbation might work better? 6. Currently SAC and its variants are used for the Pybullet environments while PPO and its variants are used for MyoSuite environments. It would be also great to provide all of those algorithms on all benchmark problems (maybe in Appendix).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

As mentioned in point 3 in the **Questions** section.

Summary

This paper proposes an exploration method that adds noise to the last linear layer (latent) of the policy network to incorporate time-dependent noise. The method is tested on locomotion tasks and evaluated on the metrics of performance and energy consumption. Evaluation is done on PPO and SAC base algorithm and an existing time-correlated baseline, gSDE.

Strengths

- Tackles an important problem of exploration in tasks with continuous action. - The proposed method is rather simple and presented well. - Thorough experiments with necessary details and hyperparameters are presented. - Performance is evaluated on return and energy conservation which is important for practical implications.

Weaknesses

- Unclear why the proposed method enabled energy-saving policy compared to another time-correlated baseline. - Energy comparison with gSDE is missing, which makes it difficult to evaluate how the noise injected in the latent layer is helping. - Few cases, the proposed method outperforms the existing method in return performance.

Questions

It is unclear why the proposed method enabled energy-saving policies compared to another time-correlated baseline. The comparison is conducted between PPO/SAC and not with gSDE (time-correlated). It is unclear how much the injected noise in the latent space contributes compared to the time-correlated component. In Figure 4, the results are mixed. It is important to understand why time correlation sometimes worsens performance. It might be interesting to consider whether the proposed task requires independent or time-correlated noise. The description of Figure 6 is currently difficult to understand. Simplifying the description would make it easier to comprehend the implications of the energy comparison. The main contribution of the proposed method lies in the addition of noise to the latent space. Therefore, it is crucial to comprehend how this component is implemented. In terms of empirical return, the performance of the proposed method is mostly similar to the baseline. However, it is important to investigate how the introduction of noise in the latent space contributes to the observed energy consumption improvements. This information would provide a better understanding of the implications of the proposed method.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

This paper proposes a new method to conduct action exploration based on the latent space representation during the reinforcement learning process. The disturbance noise in this case is based on gSDE, and the further "interval-hold" mechanism makes the noise time-correlated. The authors conduct experiments on both classic pybullet benchmarks and musculoskeletal control environments. They show in most cases, the new exploration method can give faster reward learning compared to other baselines (SAC, gSDE-SAC, PPO, gSDE-PPO).

Strengths

1. The paper is clearly written and easy to read. 2. The idea of injecting time-correlated noise is simple, novel, yet effective. 3. Extensive results comparison on a wide range of control tasks.

Weaknesses

1. Lack of exploration algorithms comparison: the authors mention many exploring techniques [20-37] in the related work section, but have not compared to any of those, nor add on top of those techniques to boost the performance. 2. Lack of RL algorithms implemented: I will be eager to see how the proposed latent exploration idea can work on other RL algorithms (TRPO, TD3, etc). 3. Lack of ablation studies to provide rationale about: (1) why learn noise magnitude rather than a fixed noise magnitude, (2) the combination of the clipping parameters

Questions

1. In Half cheetah experiment, the standard deviation for the reward curves is much smaller compared to the rest benchmarks. What could be the potential reason for it? 2. It seems like the proposed algorithm tends to introduce more uncertainty in the reward learning (an observation from Fig. 3-4) in most cases. Does that mean it is hard to get a very stable training from the proposed latent exploration? 3. A bit curious about the 10,000 hours of training. How many GPUs do you use? Do you think there is a way to reduce the workload to make it more friendly for normal AI labs in universities to conduct this kind of research?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have well discussed the limitations in section 7. For some of the limitations (training overhead), the authors put some thought and suggest new promising work directions.

Summary

The paper proposes adding time- and actuator-correlated noise instead of simple gaussian noise to improve exploration in RL.

Strengths

1. The paper is well written and easy to read. 2. The approach can be easily applied to any RL algorithm. And can be thought of as a generalization of earlier work in noise for exploration (gSDE). 3. The improvements on 4 out of 8 tasks in MyoSuite is encouraging.

Weaknesses

1. Is there any intuition as to which tasks are suitable for your method? It appears completely random that PPO outperforms everyone on handpose which is (I expect), an implicit subgoal in tasks like Baoding and Pen. Hence it is unclear why lattice underperforms on hand pose, elbow pose. 2. A nitpick: “ought” is an ambiguous word to use in the abstract and should be replaced. 3. Another nitpick: the arrows in the energy saving vs reward gain plot is not very clear. What do they indicate?

Questions

Please see weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

The paper presents an extension to the previously introduced RL exploration method gSDE called Lattice. The main idea is to add noise to the full covariance matrix (instead of only onto the diagonal) of the multivariate Gaussian used for exploration. In particular, the authors multiply state dependent noise with the weights of the last (linear) noise layer of the policy network (so that the resulting action probability distribution is known) and add this term to the weights and action dependent noise (gSDE). In this manner, the authors achieve correlation between action components. Both, the action and state dependent noise terms are sampled periodically, where the period determines the time-correlation in the noise terms.

Strengths

- thorough experimental validation on classical RL benchmarks and high dimensional muscle RL benchmarks (myosuite) - detailed implementation details and discussion about how Lattice works - paper is easy to read and understand

Weaknesses

- no comparison to other exploration methods such as [11], [18] or [30] - contribution seems rather empirical (no theoretical contribution found) - Lattice results for classical RL benchmarks (ant, hopper, walker, half cheetah, humanoid) not very convincing even in comparison to previous method (gSDE).

Questions

Please address the points in the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors appropriately addressed the limitiations of their work.

Thank you for the clarifications

My questions have been answered. I've also increased my score to 7.

Please read and respond to authors' rebuttals

Dear reviewers, Thank you for your reviews. The authors have posted their rebuttal. If you have not yet done so, please read the rebuttal and the other reviews, and comment on whether the rebuttal has addressed your comments or concerns.