Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization

Summary

The paper presents a novel method using consistency models as a policy parameterization to address online reinforcement learning. The paper first uses the dormant ratio metric to study properties of the consistency-AC method and determine that it is suitable for online RL. Then, it presents the CP3ER method, which adds an entropy regularization term to consistency AC in order to make it suitable for online RL. Finally the paper presents results showing strong performance in image-based RL benchmarks.

Strengths

- Diffusion and consistency policies are an increasingly important research area, and in particular there has been limited work training these with online reinforcement learning. This paper seems to take an important step towards that direction, and thus has the potential to be significant. - Although several other works have used similar setups to apply DP/CP to offline RL, the specific formulation the authors employ to address online RL is novel as far as I can tell. - The method overall seems principled and the results presented in the paper seem promising. - Although there are issues with section 4 which I discuss extensively in the 'Questions' section, I like what the authors are trying to do and think if the section is written by more detail it can strengthen the case for their method.

Weaknesses

- I think the biggest issue with the paper is that the story is a bit muddled. The core contribution of the paper seems to be CP3ER and the PPER regularization method. These things should be able to work for both low-dimensional and image-based observations. However, the authors specifically state that their goal is to address image-based observations. Why? Does this algorithm no longer show any benefit in these settings? The algorithm doesn't have any contributions that specifically focus on addressing image-based observations so I don't understand why that is a central focus of the evaluation. - In order to properly contextualize this algorithm I think it would be necessary to compare it to low-dim baselines like SAC and PPO on a low-dim version of the benchmarks in addition to the state-based versions. - In the introduction the authors mention that generative models like diffusion and consistency are helpful for RL because they allow the policy to represent complex behaviors. However, none of the tasks in the training suite are 'complex' enough that they couldn't be solved by a simple MLP policy. It would be nice to see perhaps some tasks where a policy that can sample multimodal actions shines above MLP-based policies. - Section 4 is confusing and there are several other minor presentation issues that I explain in the 'Questions' section

Questions

Random Questions / Comments - I find the analysis related to figure 1 to be incomplete and hard to follow. - First, it would be helpful to explain somewhere in your paper what we can infer about a neural network based on its dormant ratio, which would make this experiment easier to understand. Your paper seems to assume the reader is very familiar with Sokar et. al. but this need not be the case - How big are the datasets? Where does the online dataset come from (ie what RL algorithm, how long was it trained, how often do you update the online dataset, etc)? - Why is there a difference between online and offline dormant ratio for halfcheetah and not walker2d? And how can you draw conclusions from this experiment when you only run two environments that seem to get different results? - Figure 2 experiment is also missing a lot of details - What AC algorithm does 'Q-loss' refer to? - Where does the data for this experiment come from? - Once again, I'm having trouble understanding how to interpret these plots or how you come to your conclusion. - Figure 3 has similar issues to the last two - The results as they are look nice but I have a few comments - I think the set of baselines is incomplete. It would be nice to see a SOTA on-policy method like PPO - All of the results show normalized scores, but you never explain how they are normalized, which makes them impossible to compare to other papers Nit picks: - It would be nice if you could add one sentence in English describing what a dormant neuron is to section 3.3 for those who are not familiar with it. It would also be nice to explain why dormant ratio is important for the expressive capabilities of a network - Line 134: newtork

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors address limitations and broader impacts of their work in the appendix.

Summary

This paper proposes a novel method that employs a consistency model for policy parameterization in online visual reinforcement learning settings. It initially identifies the issues of prior consistency actor-critic methods in high-dimensional observation online RL settings through the lens of dormant ratio. Subsequently, it proposes to incorporate entropy regularization alongside a prioritized sampling strategy to regularize the policy. Empirical experiments on 21 tasks across both the DeepMind Control Suite and MetaWorld demonstrate the effectiveness of the proposed method.

Strengths

- Overall, the paper is well-organized and presented. - The analysis of the limitations of existing consistency policy training in visual RL settings is thorough and convincing. - The experimental results across a wide range of tasks are promising compared to previous state-of-the-art methods. - The paper includes a good set of ablation studies to support the design choices of the proposed algorithm.

Weaknesses

- Lack of baseline comparison in the experiments - The scope of the paper is limited to online and visual deep reinforcement learning - The proposed modification is straightforward compared with existing algorithms.

Questions

- Although CP3ER is mainly designed for online visual RL, the proposed modifications (entropy regularization, prioritized replay, Gaussian mixture distribution of value functions) do not seem to be limited to visual observation settings. It would be interesting to see how the proposed algorithm performs in state-based RL settings. - For the 8 hard DeepMind Control Suite tasks and also Metaworld tasks, the authors compare CP3ER with TACO/ALIX/DrQ-v2 but not with DrM, which was reported in the original DrM paper to achieve good empirical performance on these tasks. This comparison here is missing. - For the analysis (Figures 2, 3, 9), it would be beneficial to include the episodic return in the plots so that readers can understand the numerical performance of the policy in addition to the policy network’s dormant ratio, which is an intrinsic factor not necessarily correlated with the policy’s episodic return. - The conclusion drawn from lines 202-203, “Therefore, we can infer that visual RL will exacerbate the instability of consistency policy training caused by the Q-loss under the actor-critic framework,” does not seem convincing. Even in ordinary policy

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

NA

Summary

The paper addresses challenges in visual RL with high-dimensional state spaces, specifically focusing on improving sample efficiency and training stability. It introduces a novel method called CP3ER, which incorporates sample-based entropy regularization and prioritized proximal experience regularization to stabilize policy training and enhance sample efficiency. The proposed CP3ER achieves state-of-the-art performance in 21 tasks across the DeepMind control suite and Meta-world, demonstrating the effectiveness of applying consistency models to visual RL. The paper identifies and addresses the instability issues caused by the Q-loss in the actor-critic framework and the non-stationary distribution of online RL data. Experimental results show that CP3ER outperforms existing methods without relying on additional exploration strategies or auxiliary losses.

Strengths

1. The introduction of CP3ER, which combines consistency models with prioritized proximal experience regularization, presents a novel approach to addressing challenges in visual RL, enhancing both sample efficiency and training stability. 2. The paper provides comprehensive empirical evidence demonstrating that CP3ER achieves state-of-the-art performance across a wide range of tasks in the DeepMind control suite and Meta-world. 3. The paper provides a thorough analysis of the impact of non-stationary distributions and the actor-critic framework on consistency policy, offering insights into the underlying mechanisms that affect policy training stability.

Weaknesses

In general, the paper is well-written and easy to follow. However, there is no illustration or theoretical support for using Eq. (8) for sampling weight. It would be better for the authors to consider more theoretical analysis on the solution design.

Questions

Why using eq. (8) for sampling weight? Can other functions that shares a similar look demonstrated in Fig 4(b) also work?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations have been discussed in the paper.

Summary

The paper analyzes the problems faced by extending consistency policy to visual RL under the actor-critic framework and discovers the phenomenon of the collapse of consistency policy during training under the actor-critic framework by analyzing the dormant rate of the neural networks. The authors propose a consistency policy with prioritized proximal experience regularization (CP3ER), which employs entropy regularization to constrain policy behavior. The experiments evaluate CP3ER with recent baselines (DrQ-v2, ALIX, TACO), on 21 visual control tasks.

Strengths

- The paper is well-written and organized, and includes a thorough discussion of the relevant related works. - Sec. 4 analyzes the Consistency Actor-Critic from the perspective of dormant rates, which is very interesting. - The experimental results showcase impressive performance gains over the state-of-the-art on a wide range of robotics tasks (e.g., DM-Control, Meta-World). Ablations are also provided to delineate the impact of each modification.

Weaknesses

- The method is not directly applicable to domains with discrete action spaces. - The experiments do not compare with other consistency RL methods.

Questions

- According to [1], dormant rates in reinforcement learning and supervised learning represent different meanings. Can you explain in detail how dormant rates affect the expression ability of the consistency policy during training? For example, what does a high dormant rate indicate? What does an increasing dormant rate signify? And why? - [1] Sokar, Ghada, et al. "The dormant neuron phenomenon in deep reinforcement learning." *International Conference on Machine Learning*. PMLR, 2023. - In Sec. 4, Fig. 3, since Q-loss under the actor-critic framework will destabilize the consistency policy training, why was only the Q-loss used during the training process? - How do the computational cost and runtime compare with other baselines?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See the weaknesses and questions section.

Most concerns addressed

Thanks for running more experiments and addressing most of my concerns. I'll increase my score to a 5. My one major remaining concern is the following from my original review: > In the introduction the authors mention that generative models like diffusion and consistency are helpful for RL because they allow the policy to represent complex behaviors. However, none of the tasks in the training suite are 'complex' enough that they couldn't be solved by a simple MLP policy. It would be nice to see perhaps some tasks where a policy that can sample multimodal actions shines above MLP-based policies. The authors pointed out that their policy is indeed still an MLP which indicates to me that I may have worded my complaint in a confusing manner. Here when I say 'MLP policy' I'm referring to a setup that directly maps an input state to a deterministic action, Gaussian distribution, or something similar, and examples of algorithms that use such policies would be vanilla SAC and DDPG. On the other hand, the authors use a consistency policy which, while still parameterized by an MLP, uses the consistency model framework in order to efficiently simulate the reverse diffusion process. The authors motivate this decision in the intro when they argue that consistency policies are more capable of addressing complex tasks that would not be solvable with a Gaussian policy. However, none of the tasks in the paper are too complex to be solved by a Gaussian, so I feel that it doesn't make sense to claim this setup is more capable of such tasks.