Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient

Summary

This paper introduces DDiffPG for online reinforcement learning with multi-modal behaviour discovery. DDiffPG consists of two parts: 1) a new policy improvement method to stabilise the diffusion policy by cloning a target action; 2) a mode discovery mechanism to train mode-specific and intrinsic Q functions. In their experiments, the authors have shown that DDiffPG can achieve comparable performance with the baselines while producing multi-modal behaviours, which provides a series of benefits like avoiding mode collapse.

Strengths

The paper has introduced an interesting idea. To the best of my knowledge, this is the first work that allows diffusion policy to learn multi-modal behaviours during online RL. According to the experiments, the proposed method has produced a reasonable performance with nice multi-modal behaviours. Besides, the paper has provided nice visualisations and discussions to help understand the proposed approach.

Weaknesses

There are several main weaknesses of the paper. - The paper is hard to follow and the presentation has a certain room for improvement. - In section 3, the formal theoretical derivation of the newly introduced policy improvement objective is missing. Although it shows that this method worked empirically, it remains unclear how the resulting policy theoretically maximises the expected return in general. - I feel the paper is a bit over-claiming for certain aspects. In section 5.3, the authors claimed that DDiffPG can *overcome* local minimum issues and encourage exploration. However, the exploration comes from the use of RND when learning $Q_\mathrm{explore}$, rather than the architecture itself. In addition, it is a very strong claim that DDiffPG **overcomes** the local minimum issues. The experiments are conducted on only 8 state-based tasks, from my point of view, which is insufficient to support such a general claim. I understand that by capturing multi-modal distributions, DDiffPG allows better generalisation, but I would suggest the authors moderate the claims a bit. Minor issues: - In line 157, is this a typo? In $r^\mathrm{intr}(s, a, s’) = \max(\mathrm{novelty}(s’) - \alpha \mathrm{novelty}(s’), 0)$, should this be $r^\mathrm{intr}(s, a, s’) = \max(\mathrm{novelty}(s’) - \alpha \mathrm{novelty}(s), 0)$?

Questions

- In section 4.2, line 183, I’m not fully convinced that the RL objective can skew the policy towards a single mode. Suppose we have a Q function that nicely captures two modes. During policy improvement, let’s say we sampled a batch of trajectories that equally captures both modes, and we perform policy improvement by $\max \mathbb{E}\left[Q(s, a)\right]$. Given that our Q function already nicely captures both modes, why does such an objective cause mode collapse? Could you provide more explanations? Considering the success of DDiffPG on capturing the multi-modality in the policy space, is this really because of the way you perform policy improvement in Eqn. 1, or is it because the DDiffPG used multiple Q functions for separate modes, which just better fits the multi-modal distribution? - Regarding the use of mode-specific Q functions, it is a bit unclear to me how to stabilise the training. One issue is that during online exploration, the dataset is continuously being updated and modes are being updated. In this case, how do we fix the correspondence between the Q functions being learned and the mode? Besides, according to line 167, DDiffPG requires no predefined number of clusters, and the number of modes could be dynamic. However, we have to initialise a fixed number of Q functions. This seems a bit contradictory to me. How to define the number of Q functions during training? - It seems to me the exploration is only guaranteed by the training of $Q_\mathrm{explore}$ using RND. However, how do we balance the exploration and exploitation during RL?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

- In line 175, the definition of DTW requires task privileged information, e.g., object positions. This is a major issue and unrealistic. In more realistic scenarios, people normally have no access to such information, and as a result, this clustering approach is not applicable. - The paper only conducted experiments on 8 simple state-based environments. It is unclear how the approach could generalise to more realistic environments and tasks with visual observations. Although at the current stage, I understand that the current results reasonably demonstrate the capability of the proposed method, more realistic tasks will better support many claims made by the paper.

Summary

This paper addresses the challenges associated with employing diffusion policy in online reinforcement learning (RL), particularly the intractability of policy likelihood approximation and the bias towards a single mode. The author introduces the Deep Diffusion Policy Gradient (DDiffPG) method, which decouples exploration from exploitation. For exploration, novelty-based intrinsic motivation and hierarchical clustering are utilized to identify modes, while for exploitation, the author describes the mode-specific Q-function and a multimodal data batch. Empirical evaluations demonstrate that DDiffPG effectively masters multimodal behaviors.

Strengths

+ The application of diffusion policy for multiple modes in an online setting is promising and addresses a previously unexplored area in the literature. + The introduction of a diffusion-based policy gradient method is novel and represents a significant contribution to the field. + The work is well-motivated, and the visualization of multimodal behaviors using antmaze examples effectively enhances understanding and illustrates the practical utility of the approach.

Weaknesses

+ Several claims require additional support. For instance, the author asserts that standard exploration-exploitation strategies may easily converge towards a single mode (Lines 25-27) without providing theoretical or experimental evidence. Similar issues are present in Lines 35-36 and Lines 52-53. These statements are crucial for constructing the paper's motivation and thus require more substantial support to enhance their reliability.

Questions

+ In Lines 263-267, the author explains that DDiffPG could learn suboptimal paths. Is this statement intended to justify the suboptimal performance compared to TD3? The author suggests that this suboptimal issue can be mitigated by the mode embeddings. It would be more effective to present the best performance and use the suboptimal trajectories as ablations, specifically when blocking the optimal path, to highlight the significance of multiple trajectories. + Why does directly using the action gradient to optimize the policy lead to vanishing gradients and instability? Is this due to the large denoising steps? Including corresponding ablation studies would provide a better illustration. + Unlike the offline setting where trajectories are stable, the replay buffer with the updated Q function results in changed pairs of $(s, a^{target})$. Does training the diffusion model with a supervised framework on continually changing pairs lead to instability in learning? (Lines 126-129) + Why does DIPO, which uses the original $a$ from the buffer, not know the true outcome? I understand that the replay buffer contains the past trajectories $(s,a,r,s')$ (Lines 134-137). + Are the mode-specific Q-functions also applicable to other standard policies?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

--

Summary

This paper aims to solve online RL problems with diffusion policy. It includes 1. a diffusion policy optimization method for diffusion online training. 2. A combination of intrinsic rewards motivated skill discovery method and model-seeking Q-learning to facilitate exploration and prevent mode-collapse behavior. 3. Several self-designed environments where there might be multiple optimal solutions and thus require expressive exploration policy.

Strengths

1. The paper shows diffusion policy has a big potential in online RL because it enables multimodal exploration. 2. The self-designed environments are a good contribution to the research field by showcasing the necessity of diffusion exploration. 3. Performance clearly surpasses several baselines. follow up: The experiments basically support comments in the paper. The paper sets out to handle the single-mode exploration problem in online RL, and the self-designed environments, unlike most previous classics, allow diverse optimal behaviors and can benefit from multimodal exploration. The experiments show that the proposed method outperforms several classic baselines including some diffusion-based methods.

Weaknesses

1. The proposed diffusion training objective seems handcrafted and requires a lot of tunning. This may limit the algorithms' further application. 2. Besides the diffusion optimization methods. Other proposed techniques are more like a good combination of previous work. This indicates limited theoretical novelty. 3. Code is not provided. For this style of paper, I think code quality is essential, and a mere promise to release the code is not convincing. follow up: 1. The ablation studies are not strong enough to prove the improved performance number actually comes from multimodal exploration. I cannot be certain which part of the method works from the experiments. More visualization/empirical results/analyses should be given. 2. The formatting of the table/figure can be greatly improved. For instance, the title of Figure 4 is wrong/incomplete. Table 3/4 is referenced as the main results in the paper but only put in the appendix. 3. The diffusion optimization results also lack very strong novelty. The loss function is basically a supervised learning loss adapted for online RL, without strong convergence or policy improvement guarantee. Still, the diffusion+online RL theories are a known unsettled and hard problem, so this kind of exploration is fine and meaningful.

Questions

None

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

None

Reponse

I thank the authors for giving me detailed responses, which have resolved some of my concerns. Overall, I think the paper is above the acceptance threshold, though marginally. I thus maintain my score.