Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?

Summary

The objective of this paper is to harness the capabilities of pre-trained text-to-image models and image editing techniques in order to facilitate robot learning. This is achieved by leveraging these tools to modify the current scene towards the intended image objective.

Strengths

The paper presents a novel approach that harnesses the knowledge embedded in large pre-trained generative models to facilitate zero-shot guidance for robot learning tasks. Additionally, the paper asserts that its edited image functionality outperforms existing methods. Lastly, the paper identifies a gap in current text-conditioned image generation models.

Weaknesses

1. Figure 1 can be enhanced to effectively communicate the problem the author intends to address, while also succinctly highlighting their contribution and providing an overview of their proposed method. 2. In recent years, significant advancements in diffusion models have led to the emergence of notable works such as CACTI, GenAug, and ROSIE. These works have demonstrated considerable improvements in generative modeling and learning. However, none of these works were used as baselines in the paper, including DALL-E-Bot, which bears the closest resemblance to this work. While acknowledging technical differences, it would be valuable to compare and contrast the proposed method with these related works. 3. I appreciate the author's effort to compare various methods of image editing with their proposed approach. However, many of these methods are pretrained on different datasets, necessitating the evaluation of their performance on a comprehensive image editing dataset for a more equitable and meaningful comparison. 4. The author's inclusion of both simulation and real-world demonstrations is commendable. However, to enhance the persuasiveness of the findings, it would be beneficial to introduce slightly more complex tasks or incorporate additional distractors during evaluation to assess the robustness of the proposed method. 5. The author may also consider exploring the line of work that involves generating demonstrations using video editing techniques for robot learning, as exemplified in "Human-to-Robot Imitation in the Wild," RSS 2022.

Questions

I hope the authors can address all the questions and concerns i have in the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

1 poor

Limitations

Yes, the authors have addressed all the limitations.

Summary

Post rebuttal updating score to 5 This work proposes to use generated visual goals for RL. A diffusion model based approach is used to edit visual observations based on text prompts. The proposed image editing approach is shown to be better than prior text-based image editing approaches based on a human evaluation. Visual goals predicted using the proposed method shows improved RL performance compared to baselines that use other image editing mechanisms.

Strengths

* Leveraging pre-trained models for image editing is an interesting idea * Proposed approach seems to work better than some prior editing approaches * Some indication that the generated visual goals help learning

Weaknesses

Originality: Imagining visual goals for RL has been explored in prior work (e.g., [1] https://arxiv.org/pdf/1807.04742.pdf), which limits the originality of this work. The use of diffusion models could be new. Significance: * Experimental results do not demonstrate the need for visual goals. For instance, there are no baselines in the RL experiments which are not based on imagined goals. * Very few tasks are tested * Baseilnes are inadequate * Approach is too specific to the tasks considered and generality of approach is questionable Presentation/Clarity * Need better clarity on motivation, background on methods, problem formulation and approach overview * There needs to be a clear motivation/overview section before section 3. Technical details lack context and are difficult to follow. For instance, section 3 talks about latent diffusion models without providing any background/motivation. There needs to be a technical overview section on diffusion models. * Method description is vague and mathematical details are missing. - 3.1.1 Description of the optimization approach is vague. - 3.1.2 What is inversion and why is it necessary? - 3.1.3 Assumes prior knowledge and familiarity with diffusion models. What is x_0? What does it mean to replace attention maps? Experiments * Is the comparison on image editing fair? The proposed method is fairly specific to the tasks considered while it is compared against general purpose image generation/editing techniques such as pix2pix. * Inadequate baselines for the RL experiment: There seem to be no baselines that are not based on imagined goals/image editing.

Questions

* Many details about the setting are unclear. Need better clarity on the following. - Where do the text prompts/instructions for editing come from? - What are the inputs and the outputs of the components of the approach? - How does learning work? * Is it possible to use any quantitative metrics for image editing experiments (e.g., based on CLIP encoders)? * For the manually defined visual goals how many images were manually created? How many task instances are there altogether? * line 297 mentions ‘a certain level of prompt tuning is needed’, but such details are missing from the main text.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See above

Summary

This paper demonstrates a novel way to utilize existing large-scale text-to-image models for robot learning. Specifically, they modify existing pre-trained text-to-image models to produce visual goals for example-based reinforcement learning, before learning policies from the generated images. Experimentation in both simulation and the real world shows that the proposed modifications to pre-trained text-to-image models are able to generate higher-fidelity images than the baselines, and that this also leads to gains in downstream performance.

Strengths

- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology. - Novelty. The proposed approach seems to be a novel one which builds on ideas from the literature. - Real world experiments. The inclusion of real world experiments for both the image goal generation and the RL training make the strength of the proposed approach a lot more convincing than otherwise. - Strong results. The proposed approach outperforms baselines from prior work as well as the ablations. The qualitative results provided in Figures 4 and 5 display impressive performance of the model.

Weaknesses

- Evaluation. While both qualitative and quantitative metrics for image goal generation are included, the worry is that the qualitative examples may be cherry-picked and that the user-study done for a quantitative assessment may not be extensive enough. Would it be possible to use the standard metrics which conditional image generation papers utilize to quantitatively evaluate the proposed approach? For instance, once the ideal goal image has been generated, could the output of the proposed method be compared to using either MSE or LPIPS [1]? Furthermore, would it be possible to provide more qualitative examples? There are about three more examples per task in the appendix, but these all seem to be very similar to one another and limited in diversity. Particularly, more visualizations of the real world images would be appealing as this setting is much harder than the simulated images. - Prompt tuning. Ideally, at test-time, you are just given the current image, and a text-based goal prompt, nothing else. Line 296 says that "a certain level of prompt tuning is needed in order to achieve optimal editing performance." How much prompt tuning was done for the method? Was the same done for the baselines? [1] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.

Questions

- Qualitative Results. While the language instructions for Imagic and the proposed method are the same, ensuring a fair comparison, why are they different for InstructPix2Pix? Furthermore, why is the ablation only done for Figure 5, and not for Figure 4? - RL Experiments. Were all hyper-parameters in the RL training the same across methods? In Figure 7, the number of trajectory steps seems to vary across tasks -- why is this the case? Are the trends different if more trajectory steps were allowed?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes, the authors have adequately addressed the limitations of the work.

Summary

In this paper, authors propose a new method called LfVoid to tackle RL tasks. LfVoid can generate consistent visual goal frames through using pretrained LDM and subsequently train a discriminator to output rewards for downstream RL task. In order to improve editing consistency, DreamBooth, null-text inversion, P2P and Directed diffusion are incorporated into the pipeline.

Strengths

1. This paper is well written, clear and easy to understand. 2. An organic integration of several techiniques for controlled generation 3. Compared with existing editing methods, only the proposed LfVoid correctly modifies the desired objects and makes no obvious changes to other irrelevant things. 4. The improved generative quality subsequently enhance the performance of example-based RL.

Weaknesses

1. LDM is kind of overkilling in this benchmark. 2. Lack comparison with other RL methods leveraging goal frame generation

Questions

1. The evaluation benchmark is a little simple which more or less wastes the strong expressive power of pretrained LDM. What is the advantage of using LDM in your application compared with training a latent VAE to generate the goal frame except for less training frames? 2. As mentioned in summary, 4 techniques, including DreamBooth, null-text inversion, P2P and Directed diffusion, are incorporated but the ablation study only compares null-text P2P-DD and null-text P2P. In my opinion, the effects of DreamBooth and null-text inversion are overlapped. Can you provide any experiment to validate the improvement brought by DreamBooth token? Also, I understand the difficulty of ablation study in this work due to the lack of effective quantitative comparison. Is it possible to design some new metrics that only focus on the region of interest since the bounding boxes are provided? 3. Comparison with existing RL methods leveraging goal frame generation, for example, Goal-Aware Prediction[1], was not provided. Can you compare LfVoid with these baselines or explain why this is not meaningful? [1]. Nair, S., Savarese, S. &amp; Finn, C.. (2020). Goal-Aware Prediction: Learning to Model What Matters. <i>Proceedings of the 37th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 119:7207-7219 Available from https://proceedings.mlr.press/v119/nair20a.html.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

This paper solved the image editing problem required for goal frame generation. A solid contribution was made. I would increase my rating if my concerns are properly addressed.

Thank you

I would like to thank the authors for the clarification and response.

Thank you for the response

I appreciate the response. Clarity issues were partially addressed by the author response. I strongly recommend improving the clarity of the draft to make it more accessible to a wider audience.

Thank you

We thank the reviewer for the effort and the positive feedback.