Summary
This paper provides a systematic evaluation of the Planning abilities of a class of LLM (GPT series until the latest GPT-4), using standardized planning problems such as those provided in symbolic planning competitions. It analyzes LLM as autonomous planners, but also as heuristic planners providing suggestions to sound planners (heuristic mode). The experiments use a sophisticated prompting mechanism allowing both NL prompting and PDDL prompting. In the heuristic mode, LLM plans can be repaired via a local search planner (LPG), or use a verifier to send feedback to the LLM ('backprompting'). While LLMs score poorly as autonomous planners, they can achieve high scores (70-80%) in heuristic mode on standard benchmarks.
Post-rebuttal comments: the authors have fully answered my questions, in particular providing additional results for another LLM (BLOOM, which appears a very reasonable choice, due to how its ecosystem differs from that of the GPT family). Their commitment to release some of the evaluation resources can only enhance the contribution of this work, which is reflected in my increasing the 'contribution' score in this review. Having also considered other reviewers' comments and the author responses, I remain very positive about this paper and maintain my original score.
Strengths
The paper introduces a comprehensive evaluation method for the Planning abilities of LLM, harnessing the full methodology of traditional Planning in terms of benchmarks and reasoners. The choice of the latter is particularly appropriate to the experiments at hand, since it includes both a local planner, well-suited to Plan repair, and a validator that can send feedback by identifying gaps or flaws in the proposed plan under the 'heuristic mode'. It is fairly impressive to have automated a process previously taking place in an interactive form with a human in the loop, which was vulnerable to the Clever Hans effect, also observed in other forms of LLM reasoning [1].
The prompting mechanism is particularly sophisticated without being over-engineered, as their is a clear rationale for supporting each option and a rather elegant design starting with PDDL domains and branching out to generate NL or PDDL prompts.
In line with some claims that LLM reasoning tend to reproduce human reasoning to some extent, the choice of planning domains known to be solvable by humans is of high interest, although some of the user/human experiments have been moved back to supplementary material.
The paper is highly readable and quite systematic, and has all the elements to become a reference paper on the topic, not least for the results produced and the contrast between autonomous and heuristic modes, the latter avoiding pitfalls of interactivity or CoT limitations.
[1] Shapira, N., Levy, M., Alavi, S.H., Zhou, X., Choi, Y., Goldberg, Y., Sap, M. and Shwartz, V., 2023. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763.
Weaknesses
The paper is technically sound with very few weaknesses. Perhaps one issue is the limited number of Planning test domains used in the experiments, especially compared to [Silver et al., 2022] (ref [24] in the paper). This might be in relation to the need to explore human solutions to Planning problems as described in the supplementary material, still it could be worth justifying explicitly.
Another potential issue would be concentrating on the GPT family, as LLM may vary in their real-world knowledge depending on their training base.
The paper has similarities with the following preprint: https://arxiv.org/abs/2302.06706
This is not a major issue, either in terms of novelty or in terms of anonymity, since the submission has substantial new material and there is no direct link to the authors of the preprint. Regardless of whether the preprint is from the same authors (or a subset), it would still be appropriate to reference it in the final version of the paper.
Questions
Can any of the observations be related to the autoregressive nature of the LLM explored?
What variability in performance would you expect across LLMs (other than GPT versions)?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The conclusion section leaves little space to discuss limitations of the approach. Since CoT prompting has also been explored, it could have been interesting to discuss the proposed coupling of LLM to reasoners via PDDL exchange proposed as part of "Faithful CoT" [1].
The paper rightly identifies the potential role of the natural language semantics of predicates or operators' names in LLM's planning abilities, for which it designs various methods of obfuscation. However, further discussions would be interesting for this phenomenon reported in [2] ("semantics of the English terms used in the PDDL problems"), such semantics of PDDL contents having also been proposed as a mechanism for planning model extension and planning repair [3].
[1] Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M. and Callison-Burch, C., 2023. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379.
[2] Silver, T., Hariprasad, V., Shuttleworth, R.S., Kumar, N., Lozano-Pérez, T. and Kaelbling, L.P., 2022, November. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop. - ref [24] of the paper
[3] Porteous, J., Ferreira, J.F., Lindsay, A. and Cavazza, M., 2021. Automated narrative planning model extension. Autonomous Agents and Multi-Agent Systems, 35(2), p.19.