Library

Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models

Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning problems into the Planning Domain Definition Language (PDDL) has been proposed as a potential solution, enabling the use of automated planners. However, generating accurate PDDL files typically demands human inputs or correction, which can be time-consuming and costly. In this paper, we propose a novel approach that leverages LLMs and environment feedback to automatically generate PDDL domain and problem description files without the need for human intervention. Our method introduces an iterative refinement process that generates multiple problem PDDL candidates and progressively refines the domain PDDL based on feedback obtained from interacting with the environment. To guide the refinement process, we develop an Exploration Walk (EW) metric, which provides rich feedback signals for LLMs to update the PDDL file. We evaluate our approach on $10$ PDDL environments. We achieve an average task solve rate of 66% compared to a 29% solve rate by GPT-4's intrinsic planning with chain-of-thought prompting. Our work enables the automated modeling of planning environments using LLMs and environment feedback, eliminating the need for human intervention in the PDDL translation process and paving the way for more reliable LLM agents in challenging problems. Our code is available at https://github.com/BorealisAI/llm-pddl-planning

cs.LGComputer Science
Source

Paper

References (37)

01Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language ModelsSom Sagar, Aditya Taparia, Ransalu Senanayake2024 · International Conference on Machine Learning · 11 citations In Library
02InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task PlanningMuzhi Han, Yifeng Zhu, Song-Chun Zhu et al.2024 · Robotics: Science and Systems Conference · 65 citations In Library
03Dynamic Planning with a LLMGautier Dagan, Frank Keller, A. Lascarides2023 · arXiv.org · 64 citations In Library
04Faith and Fate: Limits of Transformers on CompositionalityNouha Dziri, Ximing Lu, Melanie Sclar et al.2023 · Neural Information Processing Systems · 665 citations In Library
05Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task PlanningL. Guan, Karthik Valmeekam, S. Sreedharan et al.2023 · Neural Information Processing Systems · 319 citations In Library
06Generalized Planning in PDDL Domains with Pretrained Large Language ModelsTom Silver, Soham Dan, Kavitha Srinivas et al.2023 · AAAI Conference on Artificial Intelligence · 216 citations In Library
07Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao et al.2023 · Neural Information Processing Systems · 4.6k citations In Library
08LLM+P: Empowering Large Language Models with Optimal Planning ProficiencyB. Liu, Yuqian Jiang, Xiaohan Zhang et al.2023 · arXiv.org · 657 citations In Library
09Teaching Large Language Models to Self-DebugXinyun Chen, Maxwell Lin, Nathanael Schärli et al.2023 · International Conference on Learning Representations · 1.3k citations In Library
10Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta et al.2023 · Neural Information Processing Systems · 4.2k citations In Library
11Sparks of Artificial General Intelligence: Early experiments with GPT-4Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan et al.2023 · arXiv.org · 4.3k citations In Library
12GPT-4 Technical ReportOpenAI Josh Achiam, Steven Adler, S. Agarwal et al.2023 · 26k citations In Library

Scroll for more · 25 remaining

Similar papers

Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task PlanningLin Guan, Karthik Valmeekam, Sarath Sreedharan et al.2023 · arXiv (Cornell University) · 316 citations In Library
LLM+P: Empowering Large Language Models with Optimal Planning ProficiencyBo Liu, Yuqian Jiang, Xiaohan Zhang et al.2023 · arXiv (Cornell University) · 653 citations In Library
Automating the Generation of Prompts for LLM-based Action Choice in PDDL PlanningKatharina Stein, Fišer, Daniel, Hoffmann, Jörg et al.2023 · arXiv (Cornell University) · 25 citations In Library
Generating Symbolic World Models via Test-time Scaling of Large Language ModelsZhouliang Yu, Yuhuan Yuan, Xiao, Tim Z. et al.2025 · arXiv (Cornell University) · 14 citations In Library
NL2Plan: Robust LLM-Driven Planning from Minimal Text DescriptionsElliot Gestrin, Marco Kuhlmann, Jendrik Seipp2024 · arXiv (Cornell University) · 29 citations In Library
Boosting Robot Intelligence in Practice: Enhancing Robot Task Planning with Large Language ModelsYisheng Zhang, Zhigang Wang, Shengmin Zhang et al.2023 In Library
Large Language Models as Planning Domain GeneratorsJames Oswald, Kavitha Srinivas, Harsha Kokel et al.2024 · Proceedings of the International Conference on Automated Planning and Scheduling In Library
Large Language Models as Planning Domain GeneratorsJames Oswald, Kavitha Srinivas, Harsha Kokel et al.2024 · arXiv (Cornell University) · 4 citations In Library
Generating Domain Models for Automated Planning from Natural Language DescriptionsGiacomo Acitelli, Andrea Marrella, Jacopo Rossi et al.2026 In Library
An Extensive Evaluation of PDDL Capabilities in off-the-shelf LLMsKaustubh Vyas, Damien Graux, Sébastien Montella et al.2025 · arXiv (Cornell University) · 3 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer cVUW5/10 · confidence 4/52024-07-03

Summary

The paper proposes an approach to leverage LLMs and environment feedback to automatically generate PDDL domain and problem description files without human intervention. They do so by an iterative refinement approach that generates multiple PDDL problem and domain candidates based on feedback obtained from the environment. The authors show their approach works experimentally in 66% of 10 PDDL domains that they have tried.

Strengths

The problem addressed in the paper is an important and interesting problem. The proposed approach with regard to the EW metric is novel and promising.

Weaknesses

Assumptions: Assumption 2 may not be realistic. Often times, people may not know exactly what is the right way to capture the domain knowledge, that is what kind of things they should have said to ensure the pre/effects/initial state all are captured. What about the case that the domain description is missing a constraint or precondition? Environment requirement: in regard to the applicability of the work, there is a dependency on the environment to do the refinement, and that also may limit the impact of the proposed solution as the environment may not always be available for all domains. Novelty: the authors claim to be the first to enable PDDL generation using LLM without human intervention. However, there exists at least two related work that does also generate PDDL domain and problem without human intervention: 

 1. Large Language Models as Planning Domain Generators ICAPS 2024 (https://github.com/IBM/NL2PDDL) 2. AUTOPLANBENCH: Automatically generating benchmarks for LLM planners from PDDL PRL 2024 (https://github. com/minecraft-saar/autoplanbench/tree/main.) The paper presentation can be improved. See the question section.

Questions

1. 66% solve rate what does it mean? Does it mean 7 out of 10 problems are solved? Also what does that mean? Is the PDDL now correct? How many problems of that domain is the different approaches now solving correctly? Can you please further clarify. 2. How do you know if you have a planning problem at hand, which can be turned into PDDL? Have you tried it on Alfworld (has a PDDL but its not ideal), or Mind2Web, or anything that does not have an existing PDDL. Maybe this relates to assumption 2. 3. With regard to the refinement how do you know when to stop? Is there a threshold on the metric that would give you signals on when to stop the refinement? 4. Can you say anything in regard to algorithm 1's soundness/correctness (and the approach in general). Also with respect to algorithm 1, how many times a LLM model should be called to come up with a reasonable PDDL domain, and a reasonable PDDL problem? Can you comment on the cost associated with that as well (any information beyond the token size would be great). 5. In the notation section 3, why is the set of all possible actions A separate from D, isn’t A always part of D. 6. Regarding planning error, what about unsolvable instances? Do you assume all instances are solvable? 7. Regarding equation 1, can you clarify that you generate multiple problem instances, but only one domain, or also refining the domain multiple times, each time generating multiple problem files? 8. Can you clarify why you need the ground truth p and d, how do you use the ground truth q to validate the answers. Also what happens if you are not provided with the ground truth domain/problem. Does this mean that even though a ground truth domain/problem is given we are going to use LLMs to generate the domain/problem? 9. Can you please comment on applicability of your approach if the PDDL environment is not known/given?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Assumption 2 may be limiting the scope and the level of impact of the work. Also what about cases that the ground truth domain/problem is not known which is in most cases when it comes to real applications. Also having to rely on the PDDL environment is also limiting the scope and impact of the work.

Reviewer cVUW2024-08-08

Can you please point to where you discuss computational complexity and soundness and completeness of your approach? I would say that the notion of human intervention can be interpreted in multiple ways: no human, human before the LLM call, human in the loop (after LLM call), etc. I would be good to distinguish your work in one of these. The related work has as you point out the assumption of predicates, but that does not mean humans are in the loop while LLM is being called, right? I am still not convinced regarding the generality of the work given the current assumptions (need for the environment, etc).

Authorsrebuttal2024-08-08

> Can you please point to where you discuss computational complexity and soundness and completeness of your approach? In response to Q7, we provide more detail on how many times the LLM is called and the time complexity of the EW metric (which we will add to our paper). In addition to that, we have mentioned in our paper (line 352) that in Table 2, using the GPT-4 model, we used 12.40 million input tokens and 8.73 million output tokens. Due to the closed-source nature of GPT-4, we are not able to compute the number of FLOPs or any other metric associated with the complexity of the LLM in our experiments beyond the token count. We would be happy to provide more information if the reviewer has particular metrics in mind. In regard to the soundness and completeness of our approach, we have formalized all the metrics and setup (section 4.1). We also provide desirable properties of our introduced EW metric (lines 259-272), and design rigorous experiments with quantifiable metrics (i.e., domain term removal experiments, and plan-not-found metric) to verify the usefulness of the EW metric (sections 4.2, 4.3, and figure 2). These are in addition to the strong results we get in Table 2 by applying our method to PDDL environments. > I would say that the notion of human intervention can be interpreted in multiple ways: no human, human before the LLM call, human in the loop (after LLM call), etc. I would be good to distinguish your work in one of these. The related work has as you point out the assumption of predicates, but that does not mean humans are in the loop while LLM is being called, right? Once the input problem descriptions are given (which is a part **benchmarking setup**, and not the method), our method requires **absolutely no human intervention**, from the very beginning of the problem/domain proposal to the very end of getting the final evaluation metrics. We will make this more clear in our paper to avoid any confusion. We should point out that there are already several papers that do not require human intervention (such as LLM+P and LLM-DP in Table 1 of our paper, as well as the work you mentioned). However, the important point is that none of the works check the "Domain translation" criteria, where the language model needs to come up with correct predicates and preconditions/effects. The assumption that "predicates are given" is too relaxing and does not pass the "Domain Translation" criteria checkmark (in Table 1 of our paper). Therefore, this does not change the fact that our work is the first to not require human intervention. > I am still not convinced regarding the generality of the work given the current assumptions (need for the environment, etc). Our main goal is to move towards fully automated planning with LLM agents, and such a goal by definition requires the interaction of the agent with the environment as one of its essential parts and most LLM Agentic workflow work needs some environment interaction. In fact, this is something that even human has to rely on in everyday scenarios (e.g. push/pull to open the door when initial mental model about which way the door opens is wrong). In the absence of an environment, relying on automated agents may not be advisable.

Reviewer cVUW2024-08-09

Re complexity, I am actually referring to a big O complexity of the approach (depending on the size of the input/dataset). See the following as examples of such complexity analysis: https://arxiv.org/html/2406.11911v2 https://arxiv.org/abs/2404.11833

Authorsrebuttal2024-08-12

We appreciate the time and effort you have dedicated to the author-reviewer discussion and thank you for bringing our attention to these relevant papers. To provide a similar big O notation analysis as the work of Katz et al., our method has an overall time complexity of $O(n_p \times n_d \times c_{max} + N )$ for generating domain and problem PDDLs (where $N$ is the number of tasks). Here, the first term is for one-time domain translation, and the second term for translating each of the $N$ tasks.

Authorsrebuttal2024-08-13

We thank you again for taking the time to review our paper. We hope to have addressed your questions and concerns. We noticed that the score was recently adjusted from 6 to 5. We would be grateful if you could give insights behind this change, and allow us to provide further explanation. If the change is related to the complexity of our method in relation to the two references above, We would like to emphasize that these papers are very recent and do not appear to be published yet. Furthermore, there is no evidence from those papers that they can solve the problem we target in this work.

Reviewer cVUW2024-08-13

1. I don't agree with the novelty point regarding human intervention. The related work I mentioned also does not have the human in the loop. The notion of human intervention can be interpreted in different ways. 2. The complexity numbers: I am not sure on what was just provided as there is no explanation what the different terms are, and don't discuss how you got there and why these complexity analysis was not part of the main paper? 3. There are assumptions that are unrealistic in a general sense, so the significance of the work is questionable. For these results, I have lowered my score.

Authorsrebuttal2024-08-14

We have previously addressed all three points that the reviewer made in the last reply above. Given the discussion period is nearing the end, it would have been helpful if the reviewer provided actual reasons instead of the same general claim that we have already rebutted, so that we could provide further explanation. We try to re-summarize our previous responses nonetheless and elaborate as best as we can. > I don't agree with the novelty point regarding human intervention. The related work I mentioned also does not have the human in the loop. The notion of human intervention can be interpreted in different ways. The related work that the reviewer referenced is responded to under “Q3: Novelty: Two other works generate PDDL domain and problem without human intervention” of https://openreview.net/forum?id=RzlCqnncQv&noteId=ofKer0JBL5. In short, the referenced papers do not solve the same problem at all, and there is no evidence that they could tackle the problem we are targeting. What we mean by no human intervention or human feedback is explained in the previous response in https://openreview.net/forum?id=RzlCqnncQv&noteId=lSXjugc9Wg , starting from the sentence “Once the input problem descriptions are given [...]” > The complexity numbers The complexity number is explained in https://openreview.net/forum?id=RzlCqnncQv&noteId=tvVozFzGuB , where the sources of the first term and second term are explained. Note that the notation is consistent with what is used in the paper text, in the main Algorithm 1 and the main result Table 1, so the result should be evident from the double loop in Algorithm 1 and the fact that the inner most portion generates $n_d$ samples from LLM on line 6 of Algo 1. To elaborate on the complexity further: (1) The complexity of domain generation is $O(n_p \times c_{max} \times n_d)$. This is because at first, $n_p$ problem candidates are generated and for each problem candidate the algorithm goes through a refinement procedure (lines 1 and 2 of Algo 1). The refinement is a tree with depth $c_{max}$ (where $c_{max}$ is the maximum number of refinement turns) (line 5), and at each level of the tree, one node is expanded with $n_d$ children (where $n_d$ is the number of domain refinement candidates) (line 6), which leads to $O(n_p \times c_{max} \times n_d)$ complexity. (2) Once the domain is ready, the complexity of task generation for $N$ tasks is $O(N)$ since for each task we only call the LLM once to get a problem translation. Therefore, the overall complexity is $O(n_p \times n_d \times c_{max} + N )$. We did not claim anything regarding computational complexity or efficiency, but only the ability to solve very challenging planning problems automatically without human feedback, so initially, it did not seem to be crucial to include the complexity, especially given that it is not hard to see from Algorithm 1. Instead, we included the actual token cost from LLMs in the paper. We will, however, add the complexity to the paper. > There are assumptions that are unrealistic in a general sense, so the significance of the work is questionable. This is already addressed in the rebuttal https://openreview.net/forum?id=RzlCqnncQv&noteId=ofKer0JBL5 and in the last section of comment in https://openreview.net/forum?id=RzlCqnncQv&noteId=lSXjugc9Wg Again it would have been helpful to have more than “in a general sense” as criticism so that we can further explain. We acknowledge that the reviewer has the right to revise their score in ways they see fit. We do respectfully disagree with the assessments made by the reviewer, but, unfortunately cannot provide a further explanation because the last round of comments are too general and the discussion period is coming to an end.

Reviewer wCSx6/10 · confidence 5/52024-07-09

Summary

The paper presents an approach that leverages LLMs to generate PDDL domain and problem files from natural language descriptions, and refine them iteratively based on environment interactions. In particular, it proposes an Exploration Walk (EW) metric that provides feedback signals to guide the iterative refinement process. In experiments, the proposed method successfully recovers PDDL files in 7 of 10 domains, outperforming LLM-based planners.

Strengths

- The paper studies the important problem of learning PDDL domains from embodied interaction for classical planning. This is a promising direction to enable long-horizon planning with formal guarantee. You may also find this recent paper [1] highly relevant. - In contrast to existing work that require human intervention, the paper boldly attempts to generate PDDL files automatically using feedabck from environment interaction. The proposed EW metric does make some sense to me. - In experiments, the method seems to be fairly capable in recovering valid PDDL files. [1] Han, Muzhi, Yifeng Zhu, Song-Chun Zhu, Ying Nian Wu, and Yuke Zhu. "InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning." RSS 2024.

Weaknesses

- My major concern is on the major contribution of the paper - the automatic mechanism that iteratively refines the generated PDDL files. - **EW does not provide a sufficient objective.** The paper presents an Exploration Walk (EW) metric to provide feedback for the refinement process. The metric measures the difference of generated PDDL domain and ground truth environment by the feasible action sequences within. While EW=1 is the necessary condition for the generated domain to be valid, it is not the sufficient condition. I agree that EW can provide guidance at the initial stage, but in the end the objective in Equation (1) should be the one to optimize to produce a valid domain. - **The exact feedback not explained enough.** The paper seems not to elaborate on the form of feedback provided to LLM for refinement - given the EW score which is a number. While the authors mention this briefly in the Appedix, I'm still don't fully understand how it works exactly. As this is the key part that makes the proposed approach possible, I would suggest the authors to provide more details in the main paper - **The effectiveness of scalar-based feedback is doubtable.** Also, given the feedback is a number that provides little information on what the exact issue is (whether it's on problem file or domain file, whether it's on a precondition / effect term or on the predicate design, and which line), I doubt whether the LLM can perform reasonable refinement. I think it's highly possible that the iterative refinement process will goto nowhere. - Another important doubt is on the problem setup - where the natual language descriptions are translated line-by-line from the ground-truth PDDL files. - **This setup is foundamentally different from what the problem of "generating PDDL" should be.** Under this setup, the challenge is no long generating PDDL files that requires exploiting environment interactions, but **translating** natural language into PDDL precisely without losing any information. More specifically, I believe the difficulty is to identify important predicates. Once the predicates are ready, the precondition & effect terms and initial & goal states should be relatively simple to be translated with GPT-4 with some prompt engineering. - **The proposed approach seems to be unaligned with the challenge**. While I think utilizing something similar to EW metric is the way to go, it doesn't aligns well with the challenge posed by this problem setup.

Questions

- What is $c_{max}$ in Algorithm 1? If it is the refinement iterations, how many iterations are used in the paper? - In experiments, the method runs 4 times and the best result is used for evaluation. I'm curious what the statistics of the 4 runs look like. Does a "magical seed" lead to good results while the others fail? - The language descriptions are generated by a GPT-4 from ground-truth PDDL files. I will expect there might be missing items or hullicination. I wonder, do you manually check and fix the generated outputs?

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

See Weakness and Questions

Reviewer zxRN7/10 · confidence 4/52024-07-13

Summary

This work talks about generating PDDL domain and problem files with LLMs. Specifically, it improves existing frameworks, particularly Guan et al. [8], in terms of increasing the degree of automation & eliminating the need for human corrective feedback. The core contribution of this work is the EW score. To compute the score, it only requires access to a set of executable plans and an executability checker (which can be either a simulator or the actual environment). The EW score is used to select sampled domain models given by the LLMs.

Strengths

- The paper is well-written, with precise and rigorous wording and formalism. - The attempt to reduce the need for human feedback in domain generation is a meaningful and useful step forward. - The introduction of the EW score not only forms the foundation of this work but also holds potential for future applications/research (e.g., for evaluation or as a heuristic)

Weaknesses

1. The domain model sampling/generation is done in a relatively simple way. The feedback message could be more informative than just indicating the inexecutable step or action in a plan. 2. Regarding the structure of the related work section, the distinction between "intrinsic reasoning" and "external reasoning" seems unclear to me, especially given that "with the assistance of basic external tools [14]" is mentioned under the "intrinsic reasoning" subsection. Also, even for the task of PDDL generation, certain degree of "intrinsic reasoning" is needed. Rather than saying "intrinsic reasoning", I guess the authors likely meant "direct plan generation." 3. It doesn’t make it clear how much knowledge is provided in the domain NL description. Clarifying on this helps readers understand whether this work leverages LLMs as a knowledge source or a "translator." -- from the examples in appendix, it seems that LLMs are used as the latter in this work (note that I am not saying translation is trivial) 4. While the EM score may be effective for selecting candidate domain models and guiding their generation, its suitability as an evaluation metric is questionable. We know there exists a "ground-truth" PDDL and our goal is to fully recover its functionality. This is a binary 0/1 problem. This is not like a generated model with 0.8 avg solve rate is more usable than a model with 0.2 avg solve rate. 5. Also, I think it's important to mention that the avg solve rate can only serve as an approximate measure of the equivalency between two domain models. A 100% avg. solve rate doesn't guarantee model equivalency (but this seems to be an easier-to-compute measure) 6. Assumption 2 is stated in a loose way. An NL description of a domain can be given at different degrees of detail (which correspond to different levels of difficulty in domain PDDL generation). 7. Line 211: I think it's better to say "as long as PDDL is expressive enough to capture the dynamic / working mechanisms of the environment" rather than "the env supports PDDL action interface." 8. It's unclear what the takeaway of Sec. 4.2 should be. Firstly, "plan-not-found" only accounts for a certain fraction of consequences caused by removing a term or predicate. Other consequences, such as producing invalid plans, can also occur. Secondly, it is well known that obtaining a valid domain model is challenging, even for humans. The authors should better explain the connection between Sec. 4.2 and the other parts of the paper. 9. The authors should give more information on the computational complexity/cost (e.g., time consumption) associated with the calculation of EW score per candidate model. ------------------------ Overall, I find this manuscript well-written, and the idea can be valuable to the community. Therefore, I am leaning towards recommending acceptance.

Questions

See the weakness section.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

See the weakness section.

Reviewer iKF86/10 · confidence 5/52024-07-13

Summary

This work presents an approach for modeling planning environments via PDDL generation using LLMs and environment feedback, without relying on human intervention. This is achieved by an Exploration Walk (EW) metric to measure domain similarity and guide domain refinement, and an iterative rectifying method that leverages LLMs to generate and refine PDDL domain and problem files. The evaluation of this method is performed alongside baselines on ten different standard planning domains from IPC.

Strengths

1. The presentation of the work is quite clear and concise. 2. The exploration walk method included in the PDDL file correction loop is a rather simple and unsophisticated way to obtain approximate scores for the domain generation process. 3. A decent amount of experimentation and analysis have been performed and stated in the work.

Weaknesses

1. Section 4.2 - demonstration of the brittleness of PDDL generation can be made more realistic such as additionally including hallucinated object identifiers or actions or symbols, which are highly probable with LLM-based code format generators. 2. The PDDL generation with LLMs approach is not as novel and the exploration walk/ environment feedback approach may not be as useful in generating completely new domains from descriptions or making custom modifications to existing domain files. The method does not really differ for rectifying problem files. 3. There are intrinsic problems with generating domain files from descriptions for domains such as Barman - where the levels of shaker and actions such as clean shot, empty shot do not translate well for LLM-based generation. More analysis and description in this line of argument are necessary.

Questions

1. This work on generating programs for planning problems [https://arxiv.org/abs/2305.11014] may also need to be cited in related work. 2. The authors may benefit from more sophisticated exploration approaches in this paper [https://arxiv.org/abs/2406.07145] that solve a different problem.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations have been addressed by the authors.

Reviewer wCSx2024-08-10

Thank you for your rebuttal. Please find my further comments below: - **About problem setup**: I understand this paper leverages environment interaction to improve the accuracy of translating natual language domain description into PDDL. In this sense, I find the title "Leveraging Environment Interaction for Automated PDDL Generation" misleading - automatically recovering a PDDL domain from interaction is fundamentally different from the goal of this work. Instead, I would suggest the authors to consider using "PDDL translation" or "PDDL generation from language descriptions" when referring to the problem. - **About EW**: I agree that the EW metric provides good feedback to further correct the translated PDDL domain file, which already has fair quality. - **About the feedback**: It makes sense to me that textual feedback is used for correcting PDDL translation. However, it seems that this aspect is missing from the method section of the paper. Highlighting the form of feedabck used is very important - it helps the readers understand the method better. - **About evaluation**: Though I understand that LLM-based translation and correction usually has large randomness, I think it's necessary to report the percentage of runs that the method produces valid PDDL files that allow planning. After the authors clarify the problem setup and the above concerns, I will be very happy to increase my score.

Authorsrebuttal2024-08-12

We are delighted that our response clarified your concerns, and we appreciate your feedback. To incorporate your feedback, we will make the following changes to our camera-ready version: - We will replace the term "PDDL generation" with "PDDL translation" in both title and the main body of the paper. - To better explain the feedback system, we will move the feedback format explanation from the appendix to the main body. This is in addition to the feedback example we provided in our rebuttal, which will be added to the appendix in the paper. - For each environment, we will report the number of seeds that succeed in generating a correct domain PDDL.

Reviewer wCSx2024-08-13

Thank you for incorporating the changes. I have raised my score to Weak Accpet. Also, I think it would be very helpful if you can briefly discuss the topic of learning PDDL domain from interaction experience [1,2,3] (without the natual language domain description), and how it is related to and different from your problem setup in the related work section. [1] Ahmetoglu, Alper, et al. "Deepsym: Deep symbol generation and rule learning for planning from unsupervised robot interaction." Journal of Artificial Intelligence Research (2022) [2] Silver, Tom, et al. "Predicate invention for bilevel planning." Proceedings of the AAAI Conference on Artificial Intelligence (2023) [3] Han, Muzhi, et al. "InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning." Robotics: Science and Systems (2024)

Reviewer zxRN2024-08-11

I acknowledge that I have read the authors' response. I find my original evaluation appropriate for the current manuscript and will therefore maintain the current score.

Reviewer iKF82024-08-14

Thanks to the authors for their response and clarifying answers. I have read the rebuttal responses and am satisfied with the proposed modifications. I find my original evaluation appropriate, but however have increased my confidence rating.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC