Compositional Foundation Models for Hierarchical Planning

Summary

The paper presents Hierarchical Planning with Foundation Models (HiP), a framework that addresses long-horizon decision-making in novel environments, using multiple modalities of data and reasoning at various levels of hierarchy. The key elements of this framework involve the use of a large language model for constructing symbolic plans, a video diffusion model to generate observation trajectory plans, and a large pre-trained robot ego-centric action model to map these plans to a robot's control space. The authors propose an iterative refinement approach for feedback incorporation, promoting a consensus among different models, without the need for large model finetuning. Experimental results on two long-horizon tabletop manipulation tasks were presented, demonstrating promising results for the proposed strategy. The authors also mention the potential for including other modalities, like touch and sound, in the future.

Strengths

The technical content of the paper appears to be sound, and the proposed framework has shown promising results in the experimental results provided. The methodology to combine language models, video diffusion models, and ego-centric robot control models into one system is comprehensive. The paper is well-structured and clearly explains the approach and the reasoning behind the decisions made. The paper's flow from problem statement to proposed solution, and finally to experimental evaluation, is logical and easy to follow.

Weaknesses

1. The baselines compared are all relatively weak. It's more appropriate to compare with foundation robotics models, such as SayCan, Gato[1], palm-e[2]. The authors argue that compared to saycan, they can generalise to new skills. However, they didn't demonstrate the generalization to new skills either. (They evaluated on unseen tasks but not new skills.) 2. While the iterative refinement approach is interesting, it could be further scrutinized to understand its limitations better, particularly concerning computational efficiency and robustness. 3. As this is mostly an experimental, please provide code to ensure reproducibility. 4. Formatting issue in line 9, 45. [1] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. In Transactions on Machine Learning Research (TMLR), November 10, 2022. [2] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378, 2023.

Questions

See above. Could the authors discuss more about the efficiency and robustness? And also compare with a few foundation robotics model to better situate the work? This paper presents a potentially significant step towards more effective long-horizon decision-making. The presented results are promising. I'm happy to raise my scores if the authors address my concerns.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors discussed the limited. They didn't comment on the broader societal impacts but I didn't see any concern.

Summary

This work presents a new approach of combining foundation models from different modalities to into hierarchical system to solve long-horizon robotics problem. The performance of the system is evaluated on two long-horizon tasks and largely out-performed other baselines.

Strengths

The approach of this work is a effective combination of previously available models. The effectiveness of iterative refinement is particularly interesting (which seems to be one of the main performance booster in this approach). The framework details are well supported by the writing and supplementary materials.

Weaknesses

I don't find weakness of this work, but I do have some questions in terms of the dataset, experiment, and analysis, see questions below.

Questions

1. What does dataset stats look like? 2. Can you provide more description rationalize the use of training data to evaluate HiP (Table 1)? 3. line 213: What is benefits of pre-training video diffusion model in terms of success rate? The description seems to be only indirectly describing the benefits? 4. It would be interesting to see some analysis on the effectiveness of subgoal generation in this approach, for example, if the granularity of the subgoal generation various, what effect does it have on the overall approach. (This is interesting to me, without this will not affect my review score.) 5. How is the computation speed look like for this approach, how fast it is to generate for an episode?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are addressed in the paper.

Summary

The paper introduces a framework called Hierarchical Planning with Foundation Models (HiP), which is designed to improve decision-making in new environments with long-term goals. HiP uses hierarchical reasoning to plan subgoals, visually reason about plans, and execute actions through visual-motor control. The framework utilizes different kinds of knowledge to support various levels of decision-making. It employs a large language model for constructing symbolic plans, a large video diffusion model for grounding the plans in the environment, and an inverse dynamics model for inferring actions from generated videos. The models are kept consistent through iterative refinement. The effectiveness of HiP is demonstrated through its application in two long-horizon table-top manipulation tasks. The paper's introduction explains that successful execution of tasks in unfamiliar environments requires reasoning at abstract, geometric, and control levels. The authors propose using multiple modalities of Internet-scale data to reason across these levels. They explain the limitations of existing models and propose HiP as a solution, which uses three different large foundation models to construct a physically executable plan for long-horizon tasks. The models are iteratively refined based on feedback from downstream models, ensuring that the final plan satisfies constraints at all levels. The authors also discuss the potential of training foundation models for videos and ego-centric actions, and the limitations of their current computational resources. They conclude by highlighting the promise of their proposed approach in long-horizon decision-making tasks and hope to inspire future work in more complex real-world tasks.

Strengths

- Hierarchical Planning with Foundation Models (HiP) leverages different modalities of knowledge, which can improve decision-making in novel environments with long-horizon goals. - The framework uses a large language model, a large video diffusion model, and an inverse dynamics model, allowing it to construct symbolic plans, ground those plans in the environment, and infer actions from generated videos. - The iterative refinement process ensures consistency between the models and enables hierarchically consistent plans that are responsive to the goal and executable given the current state and agent. - The approach is computationally efficient to train as it doesn't require any large model finetuning. - The authors demonstrate promising results on two long-horizon tabletop manipulation environments, illustrating the efficacy and adaptability of the approach.

Weaknesses

- The paper is technically similar to UniPi. The technical novelty could be potentially incremental - it adds language model to decompose the language goal to language subgoals and then apply video prediction model to it. - The work relies on foundation models for video prediction and ego-centric action prediction. Concern: Although it can be expected that video prediction model will be available in the future, but it may not be assumed to be suitable for robotics applications. Additionally, it seems to be overclaim that robotics should rely on these models. 1. The applications in robotics may need to model multiple aspects of physical environments, where “expected” video foundation models can struggle to learn How could forces be modeled in video model? How to guarantee the generated video are feasible in physics and can be executed for unseen tasks? The shown pick-and-place is a useful environment, but doesn’t support all the claims. Either the claims need to be revised or more experiments are needed. 2. It is also likely that vision (image/video) is only one of several important modalities in real-world robotic decision-making. For example, for some insertion tasks, tactile sensing could be important. 3. The computational cost of these models is not known yet at all. Although GPT4 is available for use, it only needs to transfer text over internet. If the needed models were to transfer high-quality images or even videos, how much bandwidth would they need? Is that possible to allow real-time robotic decision-making? It seems unlikely that in the near future we will have pretrained action/video models for all scenarios that can give real-time results over internet. If the goal of this paper is to explore along this direction, I think these aspects should all be considered, instead of assuming all these challenges are nonexistent and just claim the contributions. - Overall, I think the paper is a meaningful initial exploration towards this direction, but the claims of the paper should be clear about this as some assumptions are not necessarily realistic. The title “Hierarchical Planning with Foundation Models” does not necessarily match the contributions and novelties of this paper. - L68 — The step 2 “visual planning” is the key step to plan in physical level, but image-level is not necessarily the correct level of abstraction for many cases. When does it apply and when not? This seems to include lots of unnecessary details of the physical world for tasks other than just tabletop manipulation. - More general concern: The approach relies on the availability of large pretrained models in different domains. Currently, these models are only readily available in the language domain. Furthermore, as mentioned in the paper, the approach is demonstrated on smaller-scale video and ego-centric action models trained in simulation, which serve as proxies for larger pretrained models. This might limit the generalizability of the results to more complex real-world tasks.

Questions

See the weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper presents a way to leverage large foundation models to perform long-horizon hierarchical planning tasks. Specifically, the authors leverage an LLM for subgoal generation from the text instruction, a video generation model for generating a plan for each subgoal, and an action prediction model that outputs an action given the current and next observation generated by the video generation model. To make the predictions between different models consistent, the authors propose the iterative refinement procedure to refine the prediction of a model based on joint distributions. The paper includes results in simulated manipulation environments and shows improvement over prior goal-conditioned/action planner/video planner-based methods. The authors also include ablation studies to demonstrate effectiveness of various design choices in the method, such as the use of iterative refinement.

Strengths

1. The authors presents a nice way to combine three different large foundation models in the problem of hierarchical planning. The iterative refinement method seems very effective in aligning predictions of various models, which is of great significance in research that plays with using multiple foundation models. 2. The authors perform rigorous empirical evaluations via comparing to various baselines and conducting a large number of ablation studies to show the good performance of the proposed method.

Weaknesses

1. While iterative refinement method is intuitively reasonable and a natural choice for making predictions of models in the pipeline consistent, the authors use approximations of the target joint distribution in the iterative refinement step. It is unclear if the some approximations fully capture the target distribution. For example, I'm not sure if Eq. (3) is a good approximation of (2) since Eq. (3) doesn't have the reachability part and thus doesn't fully capture the feasibility of achieving the language goal. 2. It is unclear if the method can outperform previous hierarhical planning methods such as [1, 2, 3, 4, 5, 6]. Since the authors only evaluate the method in simple simulated pick-and-place tasks, I'm not sure if we need large-scale video diffusion models for video generation and rather simple latent-space world model like [1] is very likely to work. 3. Following the comment in 2, I think the authors should evaluate the method in more domains included in [1,2,3,4,5,6] and also more realistic domains such as real-world robot manipulation settings to fully demonstrate the necessity of using big video foundation models. [1] Hafner, Danijar, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. "Deep hierarchical planning from pixels." Advances in Neural Information Processing Systems 35 (2022): 26091-26104. [2] Hafner, Danijar, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. "Mastering diverse domains through world models." arXiv preprint arXiv:2301.04104 (2023). [3] Zhao, Chao, Shuai Yuan, Chunli Jiang, Junhao Cai, Hongyu Yu, Michael Yu Wang, and Qifeng Chen. "ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Tasks." IEEE Robotics and Automation Letters (2023). [4] Kujanpää, Kalle, Joni Pajarinen, and Alexander Ilin. "Hierarchical Imitation Learning with Vector Quantized Models." arXiv preprint arXiv:2301.12962 (2023). [5] Mendonca, Russell, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. "Discovering and achieving goals via world models." Advances in Neural Information Processing Systems 34 (2021): 24379-24391. [6] Tian, Stephen, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. "Model-based visual planning with self-supervised functional distances." arXiv preprint arXiv:2012.15373 (2020).

Questions

Please address the concerns raised in the section above. Update after rebuttal: I decided to raise my score after reading author’s response.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Please read and respond to rebuttal

@all reviewers, please read the author's rebuttal and respond accordingly. Thanks! Also, the authors have sent the following link to code: https://drive.google.com/drive/folders/1nEFmbfzZuVutekbipBTzhNn0COe0BKsj?usp=sharing

Reply to rebuttal

Thank you for the rebuttal! I appreciate the additional experiments and clarifications, which addresses most of my concerns. One question is why directly optimizing the correct target distribution (Eq. (2)) leads to much poorer performance. Is there something off? Overall, I’m satisfied with author’s response and would like to raise my score.