Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning

Summary

This paper proposed to apply GPT LLMs to perform PDDL model construction from natural language description and correction with feedbacks, and then use classic planners or LLM planners to construct plans from the PDDL models. Experiments were conducted in the synthetic domains to verify the proposal and found that GPT-4 can construct PPDL models with less errors and classic PDDL planner outperforms LLM planners with the constructed PPDL models.

Strengths

1. This is a new exploration of using LLMs to construct PDDL models (or symbolic world models for AI planning) 2. Experiments are conducted to compare the performance of LLM planner and classic PDDL planner on the constructed PDDL models, and find that classic planner works better on PDDL model extracted by LLMs.

Weaknesses

1. No theoretical framework is proposed to link the underlying probability models of token sequences of LLMs to the domain and action models described by formal action description languages and its formal handling (e.g. planning, validation, correction and son). Without such a theoretical formal characterization, it is difficult to evaluate the validity and applicability/boundary of this kind of approach built on top of the LLMs, prompt-engineering, chains of feedback correction and planning and so on. 2. A few experiments are given but without a systematic handling of metrics and the underlying hypotheses to be verified, and explanation to guide the application of the proposal and future research. The experimental methodology is ad-hoc.

Questions

1. Please explain the underlying models of LLMs on generating the PDDL domains and the feedback-correction mechanism; at least, explain or discuss your intuition. 2. Have you deployed the plans to real-world fetch manipulator? If you did, how does it work? 3. As your scenario is with fetch manipulator, have you try to use the LLMs to generate continuous parameters for the action operators? If not, please discuss the missing steps and future work is needed. 4. What will be the solution to connect PDDL plans to real world execution? PDDL stream might be something you want to look at; explore how LLMs approach can be further extended to work with the full stack of task-and-motion planning systems like PDDL stream (https://github.com/caelan/pddlstream). 5. Please discuss the difference between GPT-4 and GPT-3.5 rendering different results in Table 1. This will be helpful guide readers in choosing the right LLMs to apply your proposal.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

NA

Summary

This paper proposes a translation approach to using LLMs for planning. Instead of relying on LLM prompt completion to generate plans, this approach parses a written description of the domain into a valid PDDL domain description. The user can then opt to use the PDDL domain description with an automated PDDL planner, or use the domain description along with prompting to force an LLM to generate a plan that can be parsed by a plan validator. The evaluations were based upon International planning competition domains that are produced to push the limits of symbolic planning domains and are generally considered as challenging benchmarks within the symbolic planning community. The key findings of the paper can be summarized as follows: 1) Allowing LLMs to generate PDDL descriptions working in conjunction with syntax validators and human experts to create a valid PDDL description. The process was still too error-prone to completely automate without human supervision. 2) LLMs are not capable of planning even with the restricted PDDL grammar for symbolic planning domains representative of the state-of-the-art in symbolic planning community. 3) External symbolic modules that validate plans are capable of improving LLM planning performance, but is still well below the performance of a purpose-built automated planning algorithm

Strengths

The paper appropriately evaluates the capability of LLMs to plan and reason with domain sizes representative of the state-of-the-art in symbolic planning. As can be seen from table 1, the domains are significantly larger than ones seen in evaluating LLMs for planning. both in terms of number of actions and in terms of number of literals within the domain. The claims of the paper are appropriately scoped, and are well supported by the experimentation. Care was taken by the authors to explicitly characterize the difficulty of the planning domain. The key results of the paper are intuitive, in that it appears that LLMs are capable of translating information from one format (textual descriptions) to another (formal planning domain predicate and action description), but perform poorly at the actual task of planning and reasoning. Appending plan validator feedback to the prompt and replanning with an LLM is akin to backtracking in planning algorithms but is a very roundabout and computationally inefficient way of implementing search algorithms. It is essentially using the LLM output and context window as memory.

Weaknesses

Reliance on the user: The LLM is forced to generate the domain description one action at a time. This pushes the workload of identifying the right number of actions and stopping criterion onto the user. understanding whether the domain description if complete and consistent is a major aspect of generating the PDDL domain, and right now it seems to be done by the user. Explicitly stating the contribution: I would like the authors to reflect and restate what the major contribution of this paper is. Is it a prompt structure that helps LLMs consistently translate descriptions into PDDL. Or is it the definitions of the symbolic modules used to validate the planning module developed in particular to work well with LLMs. The statement of why the paper is novel, and significant seems missing from the submission, though I appreciate the value that this work brings to the table. LLMs are new on the scene, and I have seen quite a few papers in the LLM + symbolic planning that are very recent and potentially close to the NeurIPS submission deadline. I would hope that if accepted, the authors acknowledge this work and rewrite the background and related work sections accordingly [1],[2] [1] - Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J. and Stone, P., 2023. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477. [2] - Valmeekam, K., Olmo, A., Sreedharan, S. and Kambhampati, S., 2022. Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498.

Questions

Did the authors evaluate whether the LLM can generate PDDL domains of the same description at various levels of action abstraction? If so I am missing that from the draft. What is the role of the user in the proposed system? How does your proposed framework decide if the planning domain generated is adequate?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

There's been a fair amount of recent work on using LLMs to directly write plans for planning problems. In contrast, this work divides the planning process into two stages where first an LLM is used to translate natural language descriptions of actions to formal PDDL actions, then goals can be planned towards using a classic PDDL planner like FastDownward. Alternatively, the formal model can be used to validate and propose feedback on plans proposed by an LLM planner. They also explore using feedback from an automated PDDL validator (VAL) to correct things like syntax errors, as well as human feedback, both of which are ingested by the LLM which is prompted to repair the model.

Strengths

- The two stage process proposed here (first translating to a PDDL representation, then planning) is a nice setup. Unsurprisingly, when the PDDL model is correctly specified (e.g. after rounds of human feedback), the performance is much better than a setup that lacks this PDDL model construction step. - The paper flows very nicely, giving a good overview of the problem and concise summary of the method in the intro, and a nice related work and PDDL/planning background section. - The inclusion of detailed descriptions of the prompts is essential and much appreciated (and the full prompts given in the appendix) for fully understanding what the LLM is seeing. The authors do a great job of this. This kind of extensive appendix is invaluable for both replicating LLM work and understanding precisely what the inputs/outputs of the LLM are and its various failure modes. - I found a lot of the analyses / results commentary throughout the paper to be quite interesting and informative, such as the comparison between GPT-3.5-Turbo and GPT-4, or the aside about how LLMs do more "reasonable" things in their plans with respect to a human-like prior, for example leaving a knife on the counter instead of placing it on top of a toaster as the symbolic planner sometimes did. These little asides gave me a better feel for the strengths/weaknesses of LLMs in the context of planning, world model construction, etc. - The idea of translating the formal model *back* into natural language when presenting the user with errors is also a nice touch.

Weaknesses

- The human feedback seems necessary given the errors GPT-4 introduces, and presumably this method wouldn't work well at all without that step of human correction. That means the method can't be fully automated (i.e. with just VAL feedback and no human feedback), or if this is the case then it isn't discussed or evaluated in the paper. Still, this is not a huge weakness in my view – methods involving humans are good too, and perhaps this could be a first work on which future work could be built that is more automated than just humans looking through a whole PDDL model trying to spot bugs. The authors discuss this a bit in section 4.2 and appendix A.4 which are quite nicely written. - Seeing an evaluation on more domains would have been nice, but I'm not super familiar with the norms in the planning literature. What they have is enough for me to get a reasonable feel for things, so this is not a big weakness. **Overall**, this seems like a good paper that I would weakly support acceptance on. Given the heavy involvement of humans in correcting the model I don't see this as an extremely groundbreaking result, but I think it's still a moderate to high impact well written piece of research that deserves to be published and would be appreciated by a NeurIPS audience. My opinion is definitely open to revision, and I may be missing some points as discussed in Questions below.

Questions

- "GPT-3.5-Turbo produces highly noisy outputs with over 350 errors." (Line 296) I'm curious what sorts of errors are made by the model. You mention that GPT4 makes very few syntax errors (mostly reasoning errors). Is the same true of GPT-3.5-Turbo, or does it not even have syntax figured out? - I'm curious to hear more about the *newly defined predicates* (line 187) that the LLM ended up introducing in these domains. How many new predicates does it end up producing? Is it generally obvious what the "right" predicates are, such that we'd expect it to usually match the "ground truth PDDL" (line 285) exactly, or are there situations where there could be multiple reasonable choices to make for what predicates are introduced? - At the end of the human feedback process, are the models literally equivalent to the "ground truth PDDL" (line 285)? Or is there room for the LLM to make modeling choices that lead to different models (eg in the "newly defined predicates" mentioned above). This would be helpful for me understanding the 95% accuracy result – is it just that we've recovered the ground truth at this point by human corrections, and the ground truth can get 95%? To be clear, this is still a good result, as the idea of translating into a symbolic language for fast precise symbolic reasoning makes a lot of sense, I just want to see if I'm missing anything. - The exclamation point on line 345 feels a little out of place with the tone, but that's just a stylistic choice - To point out a piece of work that is quite different but seems in some ways to share in the spirit of this work – in "From Word Models to World Models" (Wong et al 2023), they similarly treat the LLM essentially as a semantic parser that translates natural language into a formal world model language (in their case, a probabilistic program) where classic tools can do most of the work. Of course, this also relates more widely to the "LLMs with tools" paradigm where LLMs call out to other libraries/simulators as in "ToolFormer" (Schick et al, 2023) or "Mind's Eye" (Liu et al, 2022). I don't personally need to see all of this in the related work, but just a suggestion if you think it might be worth drawing the connections. - You've got a missing bullet point in Fig 1 top left box - I saw in Appendix A.4 you were discussing human-usable tools for debugging PDDL models – I'm curious if you've also thought about ways that this could be fully automated? For example, starting to plan with the faulty model, hitting a failure to plan, and then somehow prompting the model with this error such that it could revise the error in the model? Would love to hear if you've thought about this.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are adequately discussed by the authors.

Summary

The submission investigates whether LLMs are able to create PDDL domains and problems description in three domains, two standard planning benchmarks, and one more challenging domain. The paper considers two related to the title: a) direct construction of the PDDL with an LLM, b) correcting errors in PDDLs. The paper also considers extracting plans with the obtained PDDL domains. The baseline is using a classical planner, that succeeds if the PDDL is correct. In c), they compare using the LLM as a planner, or using a ReAct style idea, but obtaining the prompts from the PDDL. This middle-ground scenario might be appealing for cases where the planning domain might not capture some elements of the problem, but an LLM can have additional insights. The experiments are performed using GPT-4.

Strengths

- The benchmarks used for experiments are relevant and not trivial. The household domain illustrates might be convincing for those not familiar with the planning literature. - Using the symbolic models as targets instead of solving the task using an LLM can help to bring the gap between LLM and symbolic models. - Producing a PDDL is harder than the typical implementation of LLMs plug-in. For instance, an LLM producing simple SQL might be inspected immediately, while isolated actions are not so easy to check.

Weaknesses

- As the submission says, producing more complex PDDLs might be challenging. - Both the planning benchmarks and the household domain might actually be easy for an LLM than domains in practical applications. That should be discussed further in the paper. - The results of the manual evaluation might not extrapolate to other domains, and verification by a simulator might not be available. The submission should elaborate on the issue of explainability that is largely ignored in the document. The literature on explainability for planning should be discussed, including issues like model reconciliation.

Questions

- I understand that VAL was only used for checking syntax, but VAL could also be used to validate if a plan can be executed. Was that direction studied? It seems that was easy to test as providing a plan is not much harder than the information provided to build the models. - Which part of Fig 1 corresponds to previous approaches and which part corresponds to approaches proposed in the submission?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The work offers a methodology for building PDDL models using LLM. While the study is very preliminary, the quality is not lower that largely influential recent work like ReAct. The tasks per se is more ambitious, so further investigation is needed. The paper ignores issues of explanability and other relevant tasks like state tracking. An incorrect model that partially tracks a state might still be useful. For instance, it might help to generate hypothesis for downstream tasks.

Happy with the discussion

I am satisfied with the discussion in the author's response. Apologies for missing the relevant citations. I am still unconvinced on the utility of LLM-based planners if the domain definition generated is adequate for a symbolic planner, however that is not the sole contribution of this paper. I trust that the authors will revise the introduction to include the succinct version of the stated contributions, and update the presentation of method in the paper, to improve the clarity of the presentation inline with the response here. All the edits required are merely in the text, and no additional experiments are required. I am happy to update the score accordingly.