LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents

Thank you for your thorough review and constructive feedback. We are grateful for the positive remarks on the clarity of the paper and the significance of the proposed framework for evaluating language models' planning capabilities. We respond to your comments on the weaknesses. > **Some of the proposed possible extensions on this benchmark are somewhat expected.** As you have mentioned, the two extensions we proposed do exhibit somewhat predictable outcomes. However, the significant benefit of our LoTa-Bench lies in its ability to quantitatively measure the effectiveness of fine-tuning, a widely used approach. This capability is valuable as it provides a concrete benchmark to evaluate the extent of improvement that fine-tuning can offer. In the case of replanning by feedback, our benchmark also offers valuable insight. The performance variations when using NL feedback highlight that different types of information can significantly influence outcomes. This aspect of our work facilitates easy testing of these variables. It provides a robust test environment that can shape the direction of future research in LLM-based task planning. We believe that our LoTa-Bench will encourage the community to experiment with their ideas, and this will bring interesting results. > **The feedback for replanning is based on metadata ... but this may not be always available beyond the benchmark for other downstream tasks ... Can this direction be also useful for those cases without the oracle information?** In Section 5.2, we conducted experiments on feedback and replanning. As you mentioned, we utilized oracle information to articulate the reasons for failure in the current step in natural language, which is then fed into an LLM-based planner for replanning. These experiments primarily investigated the impact of NL feedback on planning performance, under the assumption that reasons for failure are accessible. However, in scenarios where oracle information is unavailable, our approach could potentially infer the causes of step failures using sensor data from embodied agents. For instance, feedback like “(this action failed: Robot is not holding any object)” could be derived by checking if the robot’s hand sensors detect an object. Similarly, “(this action failed: put down failed)” could be inferred using vision data by object detection. This possibility represents a separate research direction. Our paper primarily focuses on evaluating the effectiveness of NL feedback when it is available, demonstrating its potential to enhance high-level planning performance. > **The authors conduct experiments based on a subset (30%) of the original datasets but do not provide some significance tests. It might be better to include them in the results (e.g., Figure 2) with different subsets.** Although we randomly selected a 30% subset, it might have some biases in selection. We fully agree with you that it would be better to include results with different subsets. We conducted additional experiments with three other subsets. Due to the limited time of this discussion period, we only included results for three models of GPT-Neo, LLaMA-1, and LLaMA-2 except for the large ones having more than 60B parameters. Our code was ready for this experiment; we only changed the random seed for subset selection on line 78 in *src/alfred/alfred_evaluator.py* in the supplementary code. The figure linked below shows the results with different subsets. https://figshare.com/s/557079abb37b1712d074 The bold lines are the same as the previous results in Figure 2. The thin lines represent additional results with different subsets. The general trends that 1) bigger models work better and 2) LLaMA series are better than GPT-Neo are the same. However, we found some differences in the results. For example, the success rates range widely from 7.21 to 12.5 for GPT-NeoX-20B according to the selection of subsets. And LLaMA-1 worked better than LLaMA-2 for some subsets. As we found subset selection affect results to some extent, we will also include full set (using 100\% samples) results with all the models in the final paper (if we have the opportunity to prepare the final paper). Specifically, we will include two figures for the subset and the full set instead of Figure 2 (a). We would like to mention that reporting results for a subset is still valuable since many researchers do not have enough computing resources, so we think it would be better to maintain a 30% subset as well so that it serves as a kind of small set for quick or affordable experiments. This subset experiment is only for ALFRED. We already used the full set for WAH-NL.

(continued) > **The used font looks a bit different from the one used in the original ICLR format. It might be better to revert to the original font.** Thank you for pointing this out. We used XeLaTeX for PDF compiling, and it gave slightly different fonts than the pdfLaTeX compiler. We corrected the compiling options and will use the corrected version when we upload the revised draft. It was not intentional, and the page count was the same. --- We appreciate your valuable feedback, and we believe that addressing these points will contribute to the overall improvement of the paper. If you have any further suggestions or remaining concerns, please do not hesitate to communicate them.

Thank you for your thorough review and constructive feedback. Your feedback provides valuable insights that we address in this comment. > **NL to API Component** In both AI2THOR and VirtualHome simulators, directly executable API calls require object instances. For example, AI2THOR uses a format like {action=“PickupObject”, objectId=1}, and VirtualHome might use “[grab] <coffeepot> (101)”. We could design the skill set to include object IDs, such as S = {“[grab] <coffeepot> (101)”, …, “[open] <fridge> (10)”}, enabling direct API calls. However, this method would substantially increase the skill set size, resulting in higher computational costs. To mitigate these issues, we formulated each skill in natural language, without object instances. The NL to API component is responsible for translating these NL skills into executable API calls, selecting the nearest object ID for execution. This approach aligns with your suggestion of using “meaningful action names.” Furthermore, employing NL skills also assists in avoiding simulator dependency, a crucial factor in maintaining system flexibility and generalizability. This strategy enables our system to adapt seamlessly across different simulation environments. > **Failure Reasons** We have analyzed the failure cases in more detail based on your suggestion. We specifically examined the detailed reasons for the failures in the ALFRED results of GPT-3 (text-davinci-003) model, which showed the highest performance. For 162 failure cases, the authors manually categorized the failure reasons. We classified the failures into six categories. * 1) Failure of action planning (e.g., performing 'Pick' instead of 'Slice' at the step where a tomato need to be sliced) * 2) Failure of object selection (e.g., grabbing a pan instead of a pot) * 3) Failure due to absence of visual grounding (e.g., attempting to grab an object inside a drawer without opening it, failure to distinguish instances when multiple objects of the same kind are present) * 4) Lack of physical understanding (e.g., failure to put down an object because there are already other objects on the table) * 5) Failure in understanding user instructions (e.g., failure to differentiate whether the user-specified 'Lamp' refers to a desk lamp or a floor lamp) * 6) Ambiguous or incorrect user instructions (e.g., referring to 'Glass' in an instruction but the actual goal condition is 'Cup'). The results of the failure analysis are presented in the table below. | Failure category | Number of failures | |-------------|---------------------| | 1. Failure of action planning | 46 | | 2. Failure of object selection | 51 | | 3. Failure due to absence of visual grounding | 21 | | 4. Lack of physical understanding | 15 | | 5. Failure in understanding user instructions | 10 | | 6. Ambiguous or incorrect user instructions | 19 | This may serve as valuable information for understanding the causes of failures in the current benchmark. We found that the main reason of failure is high-level planning failures (1 and 2), which is about 60% of the total failure. Visual grounding and physical grounding (3 and 4) also matters and it is important future research direction of considering context grounding in high-level planning as we discussed in the conclusion and limitation section. Task instructions from human users might also be incomplete, unclear, or even incorrect (5 and 6). It would open a new research direction in which robot agents interactively request clarification of an ambiguous goal.

(continued) > **Lack of tests on GPT-4** As OpenAI provides only chat-style APIs for GPT-4 [1] unlike other base models such as GPT-3 (specifically, text-davinci-003), we were unable to directly compare GPT-4 in the same configuration. However, as you suggested, we aimed to assess GPT-4's performance, the state-of-the-art model, even with some modified experimental configurations. The following describes the configuration details and results of the GPT-4 experiments that we conducted. **GPT-4 Experiment Setting** 1. Prompt GPT-4 API has three agent roles of “system,” “user,” and “assistant.” We modified the original prompt (Listing 1 in the supplementary material) to fit these roles as follows. It worked like a prompt that gives the context of a problem and the role of the robot agent. The number of examples provided for in-context learning was the same. Examples, robot skills, and task query in square brackets [] were replaced with actual texts according to the test sample and environment. ``` <System Role> You are a robot operating in a home. A human user can ask you to do various tasks and you are supposed to tell the sequence of actions you would do to accomplish your task. <User Role> Examples of human instructions and possible your (robot) answers: [The same in-context-learning examples are provided] Now please answer the sequence of actions for the input instruction. You should use one of actions of this list: [List of robot skills provided] List the actions with comma separator. Input user instruction: [Task query] ``` 2. Parsing a response An assistant (robot) agent generates action sequences for the given user instruction. Due to the unavailability of Guidance’s token selection mechanism [2] for GPT-4, we provided an admissible action list in the prompt to circumvent the issue. GPT-4 generated the answer for the given user instruction, and we extracted each step from the answer text. There was no additional post-processing tailored to GPT-4 (steps not in the provided list of the skills would not be executed correctly). **Results** | Model | ALFRED | WAH-NL | |-------------|---------------------|---------------------| | GPT-3 (text-davinci-003) | 21.36% | 40.82% | | GPT-4 | 40.38% | 34.17% | | GPT-3.5-Turbo | 10.58% | 28.82% | We tested GPT-4 and GPT-3.5-Turbo. GPT-3.5-Turbo showed lower success rates for both ALFRED and WAH-NL compared to GPT-3. GPT-4 performed well in ALFRED, showing a 40.38% success rate, a 19% improvement over GPT-3. However, in WAH-NL, GPT-4 demonstrated a lower success rate of 34.17% compared to GPT-3. Despite the limitations of the current setup, GPT-4's performance notably surpassed that of GPT-3 in ALFRED. This observation leads us to anticipate that the incorporation of Guidance’s token selection mechanism [2] for GPT-4 could yield more impressive planning capabilities. [1] https://platform.openai.com/docs/guides/text-generation/chat-completions-api \ [2] https://github.com/guidance-ai/guidance --- We appreciate your valuable feedback, and we believe that addressing these points will contribute to the overall improvement of the paper. We will revise the paper to include the failure analysis and the GPT-4 results. If you have any further suggestions or remaining concerns, please do not hesitate to communicate them.

Thank you for your valuable feedback. We appreciate your recognition of the strengths in our paper, especially the innovative benchmark development, comprehensive experimental analysis, and the valuable resources contributed to the research community. We would like to respond to your comments on the weaknesses and questions. > **[W1] Decoupling of Planning Levels** > **[Q1] How does the decoupling of high-level plans from low-level actions in your benchmark reflect on the practical deployment of task planners? Could this lead to overlooking important dynamics that only emerge when both levels of planning are considered together?** Decomposing complex long-horizon tasks into multiple levels of planning is a widely used strategy. Consistent with this approach, our paper decouples high-level plans from low-level controls as you mentioned. We intentionally focused on high-level planning, particularly employing an LLM-based method. Our approach offers enhanced generalizability compared to end-to-end methods, which typically require extensive, task-specific data collection and training [1, 2, 3]. Our methodology is inherently more flexible, capable of adjusting to new environments without the need for additional data collection or exhaustive training. Furthermore, we recognize the importance of ensuring that our high-level planning seamlessly integrates with low-level controllers through natural language. Thanks to recent advancements in object navigation [4] and language-conditioned manipulation [5, 6], the NL steps generated by LLM-based task planners can be effectively executed. [1] Learning Neuro-Symbolic Skills for Bilevel Planning \ [2] Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning \ [3] Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation \ [4] A Survey of Embodied AI: From Simulators to Research Tasks \ [5] BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning \ [6] RT-1: Robotics Transformer for Real-World Control at Scale > **[W2] Lack of Visual Understanding** In response to this concern, we wish to clarify that our LoTa-Bench is strategically designed to concentrate on high-level task planning, distinct from vision-based low-level control. The benchmarks’s primary goal is to provide a clear and focused evaluation of language-oriented high-level task planning performance, thus intentionally minimizing potential confounding influences from low-level controls. However, we agree with your point regarding the importance of visual understanding. We are actively considering potential expansions to LoTa-Bench that would integrate visual information. This extension is designed not to exploring visual understanding for low-level control purposes but to enrich the capabilities of high-level planning. Potential directions include leveraging natural language feedback informed by visual information or integrating visual affordances into the planning process. Our objective with these future developments is to effectively bridge the gap between language-oriented task planning and visual perception. > **[W3] Domain Gap in Simulation** > **[Q3] How do you plan to address the domain gap between simulation and real-world application, and do you have strategies for dealing with unrealistic assumptions like the one mentioned in ALFRED about objects being cleaned once put into water?** As you pointed out, since LoTa-Bench evaluates planning performance in simulator environments, there is a domain gap with real world. The most prominent gap is ALFRED's assumption about cleaning. While other simulation tasks, such as picking and placing, heating using a microwave, and washing using a dishwasher, exhibit a smaller gap from real-world scenarios at a high-level planning stage. In response to this, one potential approach to expand the proposed benchmark is to enhance the AI2THOR simulator used in ALFRED to reduce the domain gap. It may become more realistic by adding conditions, such as requiring the target object to be touched with a sponge, particularly addressing unrealistic gaps like cleaning, and incorporating this into the object status update logic in the simulator. Additionally, cleaning using a dishwasher, as supported in WAH-NL, could also be added. For example, instead of simply placing a fork in the water, we can add steps, placing fork inside a dishwasher, close the dishwasher, and turn on the machine.

(Continued) > **[W4] Another limitation of the paper is the absence of an in-depth investigation into several influential factors that affect the performance of task planners. ... Addressing these factors is crucial for a thorough understanding of how to optimize task planners for real-world applications.** Our paper indeed addresses each of the factors you have mentioned, using our proposed LoTa-Bench framework. The details are described in Section 5 and 6 of our manuscript. * **1) Impact of types and sizes of pre-trained language models** In our work, we utilized the LoTa-Bench framework to evaluate and analyze the task planning performance of various pre-trained LLMs, including GPT [7], GPT-Neo series [8, 9, 10], OPT [11], MPT [12], LLaMA 1 [13], and LLaMA 2 [14], and their sizes. Our experimental results, summarized in Figure 2 of the paper, show that task planning performance generally increases with the size of the language model. Specifically, GPT-3 175B demonstrates the best performance on both ALFRED and WAH-NL. Further details are provided in: * **Section 5.2**: “We evaluated the planning performance of the baseline planner described in Section 3. … Instruction- and chat-tuned models (dashed lines in Figure 2) did not perform better than their base models.” and **Figure 2** * **2) Influence of the number of in-context examples** We explored the influence of the number of in-context examples using LLaMA 2 13B model. Our tests varied the number of examples from 0 to 30 in ALFRED and from 0 to 15 in WAH-NL. The results, depicted in Figure 3 of the paper, indicates that planning performance improves with an increased number of examples. More information can be found in: * **Section 5.2**: “We have investigated the impact of the number of examples in prompt with LLaMA 2 13B model that supports a longer context length of 4096. … (with an average of 13.61 and a standard deviation of 3.22) for the LLaMA 2 13B model.” and **Figure 3** * **3) Strategy for selecting in-context examples** We introduced three strategies (Random Sampling, Task-Specific Sampling, Semantic Similarity Sampling) for selecting in-context examples from the train set in Section 6.1 and investigated their impact on planning performance. Notably, Semantic Similarity Sampling significantly enhances planning performance. More details are available in Section 6.1 and Figure 5. * **4) Planners’ ability to replan based on natural language feedback** In Section 6.2, we discuss the effect of feedback and replanning. We observed that providing NL feedback about the reasons for action failure aids task planners in replanning when the size of language model is large. Further insights are available in Section 6.1 and Table 1. * **5) Impact of fine-tuning on train set** Our investigation into the impact of fine-tuning on the train set is discussed in Section 6.3. We employed LoRA [15] for fine-tuning LLMs, which resulted in significant performance improvements. This is elaborated upon in: Section 6.2 and Figure 6. We would like to emphasize the critical role of our LoTa-Bench framework in facilitating these comprehensive evaluations. Without such a benchmark suite, conducting in-depth analyses of task planner performance across the factors you have highlighted would be more challenging. Our framework not only enables these detailed evaluations but also provides substantial support in advancing the field of task planning. [7] Language Models are Few-Shot Learners \ [8] GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow \ [9] GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model \ [10] Gpt-neox-20b: An open-source autoregressive language model \ [11] OPT: Open Pre-trained Transformer Language Models \ [12] Introducing mpt-7b: A new standard for open-source, commercially usable llms \ [13] LLaMA: Open and Efficient Foundation Language Models\ [14] Llama 2: Open Foundation and Fine-Tuned Chat Models \ [15] LoRA: Low-Rank Adaptation of Large Language Models > **[Q2] Given that AI2THOR and VirtualHome lack diversity to reflect real-world environments, could you elaborate on your plans to enhance these simulators or to incorporate additional platforms that might offer more realistic and diverse scenarios?** Our evaluation framework is designed to be extendable with different dataset and simulator pairs. For successful integration, the dataset must include natural language instructions and goal conditions, while the simulator should support high-level APIs. To enrich our framework with broader range of scenarios, we are actively considering the integration of additional platforms. Currently, we are examining the potential of incorporating Behavior-1K and OmniGibson simulator [16], or creating custom datasets within the NVIDIA Isaac Sim simulator. [16] BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation

(Continued) > **[Q4] Can you discuss the potential development directions for evaluation frameworks that can handle the complexity of LLM-based task planning beyond simple tabletop manipulation tasks, as mentioned in the current limitation of the field?** In order to address the complexity of LLM-based task planning beyond simple tabletop manipulation tasks, it is essential to develop datasets and simulators for a variety of scenarios that focus on long-horizon tasks, similar to LoTa-Bench. The datasets we used in the present paper have 92 different home scenes in ALFRED (valid set) and 100 in WAH-NL (test set). The environment spans diverse rooms including bedrooms, kitchens, bathrooms, and living rooms. And we would like to mention that our benchmarking includes long-horizon tasks, which requires more than 25 action steps. > **Minor one: Missing references** Thank you for your pointing out these references. The first paper introduced a method for encoding an environment in the form of an object table to provide context to a language model-based planner. The second paper enhances LLM-based task planner with additional modules, eliminating irrelevant objects and receptacles from observation and tracking the completion of each step. Both works contribute to the field of LLM-based task planning by focusing on integrating contextual information to improve planning performance. We plan to add both papers to the Related Work section of our manuscript as follows: “Moreover, integrating context into LLM-based task planners has been shown to enhance planning efficacy (Huang et al., 2023; Yao et al., 2023; Chen et al., 2023; Lin et al., 2023; Wu et al., 2023)” --- We thank you again for your valuable feedback, and we believe that addressing these points will contribute to clarifying the contribution of the paper and guiding the direction of our research expansion. If you have any further suggestions or concerns, feel free to communicate them.

Thank you for your valuable feedback. We would like to respond to your comments on the weaknesses and questions. > **The primary contribution of the paper seems limited to the integration of several existing benchmarks into a single platform. ... The paper does not make it clear how LoTa-Bench provides additional value over these existing benchmarks.** First, we clarify the differences between LoTa-Bench and existing benchmarks. * **1) LoTa-Bench vs. ALFRED + AI2THOR** ALFRED [1] built on AI2THOR [2] has both high-level and low-level instructions. One example is as follows. ``` Task instruction: "Move a towel to the sink" Step-by-step action instruction: 1. Move to the far left cupboard under the far left sink 2. Open the cupboard and remove the towel inside, shut the cupboard 3. Carry the towel to the left sink above the cupboard 4. Place the towel in the sink ``` As low-level instructions in natural language are given, the previous studies [3, 4] on ALFRED mostly focused on translating step instructions into robot actions with visual understanding, which overlooks the importance of high-level planning. Thus, we proposed LoTa-Bench designed to focus on high-level task planning by using only task instructions; no step-by-step instructions were given and no training on the dataset unlike previous studies to validate generalizability of the high-level task planner. Although we did not augment the ALFRED dataset itself, we utilized the dataset in a new way to propose a new benchmark problem to help the research community validate new task planning methods more easily and objectively. * **2) LoTa-Bench vs. ActivityPrograms \& Watch-and-Help + VirtualHome** The ActivityPrograms dataset [5] provides natural language instructions, but it lacks goal conditions. Some language-oriented task planning studies that used this dataset relied on human evaluation for quantitative performance evaluation [6, 7]. LoTa-Bench, however, includes datasets with goal conditions, enabling automated quantitative performance measurement. The Watch-and-Help dataset [8] incorporates goal conditions but not NL instructions, limiting its use for language-oriented task planning. LoTa-Bench adapts the Watch-and-Help dataset for language-oriented task planning by modifying the goal conditions for autonomous agents and enriching it with NL instructions that we collected from crowdworkers. Beyond providing benchmark suites, our comprehensive experimental analysis illuminates factors influencing LLM-based task planner performance, such as model size, type, and the number of in-context examples. We also investigate performance-enhancing extensions for LLM-based task planners within LoTa-Bench, including in-context example selection, feedback and replanning, and fine-tuning. Together, we believe that these contributions uniquely position LoTa-Bench as a valuable asset in language-oriented task planning research. [1] Alfred: A benchmark for interpreting grounded instructions for everyday tasks \ [2] Ai2-thor: An interactive 3d environment for visual ai \ [3] Episodic Transformer for Vision-and-Language Navigation \ [4] Agent with the big picture: Perceiving surroundings for interactive instruction following \ [5] Virtualhome: Simulating household activities via programs. \ [6] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents \ [7] Parsel : Algorithmic Reasoning with Language Models by Composing Decompositions \ [8] Watch-and-help: A challenge for social perception and human-ai collaboration

(countinued) > **Why was there no fine-tuning conducted with the WAH-NL dataset?** In our initial approach, the primary goal of fine-tuning was to demonstrate the capability of our benchmark in facilitating a broad range of experiments for LLM-based planning. Consequently, we chose to present fine-tuning results solely within the ALFRED domain, considering it to be sufficiently illustrative of our benchmark's efficacy. Additionally, the WAH-NL domain offered a limited dataset of only 250 instances for fine-tuning, which we initially deemed inadequate for achieving significant performance improvements. Nevertheless, inspired by your query, we have now undertaken a fine-tuning experiment using the WAH-NL dataset. This dataset comprises 250 training instances, and we maintained consistency in hyper-parameters and conditions as per our ALFRED dataset experiments. The outcomes are detailed in the table below. **Table 1. Plan execution success rates of planners fine-tuned on the WAH-NL dataset. δ denotes changes in success rates with and without fine-tuning (expressed in percentage points, pp).** | | Baseline Success(%) | Fine-tuned Success (%) | δ (pp) | |-------------|---------------------|------------------------|-------------| | LLaMA 1 7B | 28.65 | 21.36 | -7.34 | | LLaMA 1 13B | 23.70 | 29.47 | +5.77 | | LLaMA 1 30B | 32.22 | 33.78 | +1.56 | Although the improvements were modest and not consistent across all models, fine-tuning with the LLaMA 1 13B and 30B models did exhibit some performance enhancements (5.77 and 1.56 percent points). The entire process, excluding the time spent on fine-tuning the models, took less than 7 hours, and the evaluation was fully automated using the proposed LoTa-Bench. We are confident that our benchmark suite can significantly accelerate research on various aspects of LLM-based planners, offering an efficient and versatile evaluation tool for the research community. --- We thank you again for your valuable feedback, and we believe that addressing these points will contribute to clarifying the contribution of the paper. If you have any further suggestions or concerns, feel free to communicate them.