Text-Aware Diffusion for Policy Learning

Summary

The paper proposed a reward generating pipeline leveraging a text-conditioned diffusion model with a text prompt to master RL tasks described by the prompt. The pipeline compares the difference of the generated image with/without the prompt and the original image to calculate a dense reward, which also makes sense intuitively.

Strengths

- The authors cleverly use a conditional diffusion model to estimate the alignment level of an image to the text prompt with a hypothesis that diffusion models capture naturalness and world knowledge. - The authors expand the proposed method from image-text space to video-text space. - As it is hard to quantitatively evaluate the performance of benchmarks, the paper conducted user studies to compare benchmarks by human volunteers.

Weaknesses

- The proposed method requires more hyperparameter-tuning of the noise level and the overall reward function. Thus, more investigation is needed before training is conducted, such as Appendix A. - The proposed method heavily relies on the capability of the external diffusion model to obtain world knowledge. It is a natural sin due to the complexity of the tasks the paper is solving. - Another minor weakness is that some details of the user study are not presented, such as the variance of ✔ and ✘ and scores (instead of giving the majority and avg).

Questions

- Is $\hat{\epsilon}_\phi$ a neural network (Sec 3.1) or a noise (Fig.3)? - What is the cons of replacing the diffusion model in your method with a CLIP-like model to access the alignment level of image and text? One I can think of is the naturalness cannot be calculated in such case. - Why is VLM-RM outperforming the proposed method in the standing Humanoid environment? Is it due to the training method of VLM-RM or because it is easier? - Is it possible to give a concrete example of the failure mode mentioned in the limitation section? - It would be interesting if the authors can try difficult tasks that even the diffusion model does not know how to generate and see the corresponding results of the RL training and evaluation.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

See weakness 1. The paper concluded most of its limitation in the main text and Appendix.

Summary

The paper introduces Text-Aware Diffusion for Policy Learning (TADPoLe), a method for reinforcement learning that leverages pretrained text-conditioned diffusion models to compute dense reward signals. This approach allows agents to learn text-aligned behaviors without the need for expert demonstrations or handcrafted reward functions. The authors demonstrate the effectiveness of TADPoLe in various environments, including Humanoid, Dog, and Meta-World, achieving zero-shot policy learning with natural language inputs.

Strengths

The paper provides a novel approach to reinforcement learning by using large-scale pretrained generative models to generate reward signals based on natural language descriptions. This method removes the need for manually crafted reward functions, which are often a bottleneck in RL tasks. The approach is well-motivated, leveraging the rich priors encoded in generative models trained on vast datasets. The experimental results show the ability of TADPoLe to learn diverse behaviors across different environments and tasks.

Weaknesses

1. Presentation Issues: The paper's presentation is problematic. The integration of components such as the 'symlog' of the reward function and the 'noise level range' is not clearly explained. It is difficult to understand the necessity and utility of these components and how much performance improvement depends on them. Additionally, the paper lacks clarity on the source of ground-truth for rendered subsequent images and whether the method requires pre-existing well-rendered videos for each environment for TADPoLe training. 2. Alignment with Motivation: The motivation to leverage text-conditioned diffusion models for reward generation is clear; however, the demand for well-rendered videos for each environment does not fully align with the goal of making reinforcement learning more practical and scalable. Some environments donot have existing well-performed video for the agent, so will this method require for a pre-trained policy to collect the video data? 3. Experimental Insufficiency: The experiments are not comprehensive. There is a need to expand the range of tasks to provide a more thorough evaluation of the method. Additionally, the paper had better include comparisons with the Diffusion-Reward method [1] to highlight the advantages of TADPoLe. 4. The ablation study is also insufficient. It should further explore the design of the 'symlog' component and the selection of the two weights, as these operations lack detailed justification. This analysis is crucial to understand the contribution of each part to the overall performance and to provide a clearer justification for their inclusion. 5. Overclaim. The writing of the paper is dense and lacks clarity in several key areas. The explanations of the methodology and components are not sufficiently detailed, making it challenging for readers to grasp the full picture of how TADPoLe functions and why certain design choices were made. Improving the clarity and coherence of the writing would significantly enhance the paper's readability and accessibility. [1] Huang, T., Jiang, G., Ze, Y., & Xu, H. (2023). Diffusion Reward: Learning Rewards via Conditional Video Diffusion. arXiv preprint arXiv:2312.14134.

Questions

1. Clarification of Component Integration: Why is the 'symlog' transformation necessary? How does it and 'noise level range' specifically contribute to the performance improvements observed? More detailed explanations and justifications are needed. 2. Source of Ground-Truth: Where do the ground-truth rendered subsequent images come from? Does the method rely on pre-existing well-rendered videos for each environment? This aspect should be clarified to understand the method's applicability. 3. Experimental Expansion: To provide a more comprehensive evaluation, expand the range of tasks and include comparisons with the Diffusion-Reward method. This will help to position TADPoLe's performance in a broader context. 4. Ablation Study: Conduct a more detailed ablation study to isolate the effects of the 'symlog' transformation and the specific weightings used. This will help to understand the contribution of each part to the overall performance and provide a clearer justification for their inclusion. 5. Scalability and Practicality: Address the scalability issue regarding the need for well-rendered videos in environments lacking existing well-performed videos. How does the method handle such cases, and what are the implications for its practicality and scalability?

Rating

4

Confidence

4

Soundness

3

Presentation

1

Contribution

2

Limitations

The author has addressed a limitation. It's also recommended to add the unsolvable questions to the limitations.

Summary

The paper presents Text-Aware Diffusion for Policy Learning (TADPoLe), a framework that leverages pretrained text-conditioned diffusion models to generate dense, zero-shot reward signals for policy learning in reinforcement learning tasks. The approach aims to address the limitations of manually designed reward functions by utilizing large-scale generative models to encode rich priors that guide policies in a natural and text-aligned manner. Experiments demonstrate TADPoLe’s effectiveness in learning policies for novel goals and continuous locomotion behaviors in various environments, including humanoid, dog, and robotic manipulation tasks, without ground-truth rewards or expert demonstrations.

Strengths

**Strengths:** - **Innovative Reward Generation:** TADPoLe introduces a novel approach to reward signal generation using pretrained diffusion models, reducing the need for manually crafted reward functions. - **Zero-Shot Learning:** The framework supports zero-shot policy learning, enabling the agent to learn new tasks and behaviors from natural language descriptions without prior demonstrations. - **Diverse Applications:** Demonstrates versatility across different environments and tasks, including humanoid and dog locomotion, and robotic manipulation in the Meta-World environment. - **Human Evaluation:** Qualitative assessments show that the policies learned by TADPoLe are perceived as more natural and aligned with the provided text prompts by human evaluators.

Weaknesses

**Weaknesses:** - **Evaluation Metrics:** While the paper provides qualitative evaluations, the reliance on human judgment for assessing the naturalness and alignment of behaviors could introduce subjectivity. - **Scalability of Text Prompts:** The approach may struggle with very complex or ambiguous text prompts, and the extent to which it can handle highly detailed or context-specific instructions is not fully explored. - **Comparative Baselines:** The comparisons with other text-to-reward methods are limited, and it would be beneficial to include more diverse baselines to better understand the relative performance of TADPoLe. - **Computational Overhead:** The approach involves significant computational overhead due to the use of large-scale diffusion models, which could limit its practicality in resource-constrained environments.

Questions

### Questions 1. **Reward Scaling:** How does TADPoLe handle the scaling of reward signals across different tasks and environments to ensure consistent policy learning? 2. **Text Prompt Complexity:** Can the method be extended to handle more complex and detailed text prompts? What are the limitations in terms of prompt complexity and specificity? 3. **Long-Term Dependencies:** How does TADPoLe perform in tasks requiring long-term planning and dependencies? Are there any mechanisms to ensure temporal coherence in learned behaviors? 4. **Generalization:** How well does the approach generalize to unseen environments or tasks that are significantly different from the training scenarios?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Same as the above section.

Summary

The paper introduces Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a large-scale pretrained text-conditioned diffusion model to provide zero-shot reward signals for training agents without expert demonstrations or manually designed reward functions. TADPoLe enables agents to learn behaviors and achieve goals specified by natural language in various tasks, demonstrating its effectiveness.

Strengths

1. The idea is straightforward and easy to follow; the authors have adopted a clear approach to present their insights. 2. Using text-conditioned diffusion models to provide rewards is novel.

Weaknesses

1. Although the text-aware diffusion reward is novel, both text-aware rewards and diffusion rewards have been proposed by prior works. The paper lacks an apple-to-apple comparison, making it difficult to discern the specific advantages of using diffusion models. 2. I find the experiments to be quite limited. For example, the authors only use TD-MPC as the algorithm backbone, which is just one model-based RL algorithm and does not have a significant advantage in visual RL. Additionally, the curve shown in Figure 5 is quite odd—why not place the baseline and proposed method on the same graph and include curves for other prompts? Furthermore, the comparisons in the Metaworld experiments are minimal, making the experimental results less convincing. 3. The absence of real-world experiments makes it hard to assess the paper's contribution to the community. 4. The authors use a diffusion model to generate rewards at each step, but there is no detailed analysis of the computational cost and its impact on speed.

Questions

1. Even with sparse rewards in MetaWorld, the state-of-the-art results are better than those presented by the authors. 2. Is the naturalness in Table 2 determined through user surveys? I believe that videos would provide more compelling evidence. 3. How do different lengths and forms of prompts affect the rewards? What impact would slight adjustments to the prompts during training have?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors discuss limitations in the paper about how to control the weight of each individual word in the prompt. However, I believe that the primary limitations lie in the efficiency of the diffusion model. This paper has many aspects that require further discussion and improvement.

Response to rebuttal

Dear Authors, Thank you for your detailed rebuttal and the clarifications provided. I appreciate your effort in addressing my concerns, especially the clarification that there is no need for expert data or pre-existing well-rendered videos. This has resolved a significant part of my concerns, and I am inclined to adjust my rating to 4 in recognition of it. However, I still have some remaining concerns that I believe are important to address: Symlog Regularization: The additional results in Table A6 from your rebuttal indeed highlight the significance of the symlog normalization. This brings up a question regarding the flexibility and robustness of your method—have you considered or experimented with alternative regularization methods? It would be insightful to see if other techniques could provide comparable or even improved performance, which would further strengthen the generalizability of your approach. Experimental Rigor: While I appreciate the addition of Table A8, which compares your method with an adapted version of Diffusion-Reward, I agree with the general sentiment among reviewers that the experimental evaluation remains somewhat weak. The results from MetaWorld, where most tasks either achieve 100% or 0% success rates, suggest a potential limitation in the diversity and complexity of the tasks. Incorporating more challenging environments, such as those involving interactive tasks with Franka-Kitchen or the dexterous manipulation tasks with Adroit, would provide a more convincing demonstration of your method's capabilities. This would be crucial for further validating the practicality and robustness of TADPoLe. Presentation: Finally, I recommend revisiting the presentation of your paper. Specifically, Figure 3 could be redesigned to locate in a line with Figure 2, allowing for space to provide a more detailed and visually appealing presentation of your experimental outcomes. Best regards.