Egocentric Planning for Scalable Embodied Task Achievement

Summary

The paper introduces Egocentric Planning, alternating exploration and task solving symbolic planning for long-horizon object-oriented POMDP in environments with deterministic action effects. The presented method is used as the planner in a hybrid agent, SOTA in the 2022 ALFRED benchmark, with neural perception. The agent is based on a previous successful design (FILM), with semantic SLAM in the latter replaced by a graph representing the current knowledge about the scene, besides the novel open loop planning algorithm exploiting this graph. The experimental results show a considerable improvement over FILM at the cost of longer trajectories required to succeed due to the time required to gather information about the scene, besides error analysis and ablations. An exciting result is the generalization to new task types not present in training.

Strengths

- The proposed method enables out-of-the-box generalization to new task types (within the same set of objects and relationships). - The graph representation and planner algorithm work as a tailored alternative to semantic SLAM for object-oriented POMDPs with remarkable performance.

Weaknesses

- The presented approach is effective under the assumption of deterministic action effects, but in real-world usage this should not generally apply. The paper consequentially discusses some possible ways to overcome this limitation, but also claims some better performing alternatives should be considered as baselines rather than comprehensive solutions, since they are limited in scope. I'd encourage rephrasing that discussion, which as is could be interpreted as unfair to the alternatives. - The need for an exploration phase, when the egocentric planner is in principle designed to determine when to explore, seems a little ad hoc. I wish there was a clearer explanation why this cannot be provided by the main algorithm.

Questions

1. How could the proposed framework be extended for multiple agents interacting with the environment? 2. Related to one of the weaknesses, what is missing in the egocentric planner to be capable of successfully exploring the environment until enough information is gathered to solve the task?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

I think the paper makes a good job describing limitations of the proposed method, but, as mentioned in Weaknesses, I think the judgment of better performing alternatives as baselines, given the acknowledged limitations of the proposed method, could be seen as unfair, so I think an improved contextualization is needed.

Summary

The authors propose an approach combining symbolic planning and object-oriented POMDPs for symbolic planning, which gets extremely strong performance on the ALFRED benchmark and won the CVPR ALFRED challenge. Their approach uses PDDL, but extends it with a set of exploration-focused actions. They use a combination of explicit knowledge and heuristics (exploring close to a seen object for example) that are learned from data. They explore for 500 steps, using this to build a spatial-semantic graph (instead of an explicit map like many previous methods). They then use this with an off-the-shelf task planner to choose which actions to execute.

Strengths

- Building a spatial semantic graph seems like a better (more scalable) way of solving ALFRED tasks than explicit maps - Strong performance on a well-respected benchmark - A lot of great ideas, and some of the explanation is very good - Great to see a *new* approach to solving ALFRED, not just building off of FILM/HLSM

Weaknesses

- The name "egocentric planning" doesn't make a lot of sense to me; everything in ALFRED is going to be some manner of egocentric planning - Not clear how general this is - lots of engineering in the planning domain. More analysis would help here; ideally the authors could run on a different domain (OVMM might be one - ovmm.github.io), but there really aren't good options with strong existing baselines like ALFRED. Instead maybe they could describe in more detail how it would be applied to other domains. - Writing could be improved. Several typos (Apporach --> Approach), IER (exploration algorithm) not being referenced by name in text. - It's not really clear this is a *learning* contribution - learning components seem minor. I think this is ok, because it's still a useful result on a learning problem, but I could see the argument against it. - Very engineering heavy.

Questions

- How is the spatial graph constructed? It seems like a lot of the detail here is lacking. - How important are the 500 exploration steps? Seems like it should be able to work well without this, if it's really a good POMDP planner. - Construction of the initial state was also a bit unclear to me.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Only applied to one benchmark - Need for 500 exploration steps at the beginning seems weird - Very dependent on perception models - perfect detection, depth, etc., which limits application to other domains.

Summary

The paper presents a modular approach that combines symbolic planning and object-centric Cost POMDP for solving ALFRED tasks. The proposed method demonstrates improvements compared to previous end-to-end and modular approaches, such as FiLM. Unlike methods like FILM, HLSM, and Prompter, EPA utilizes a semantic graph representation instead of a top-down 2D map. The method incorporates an initial phase of 500 exploration steps to gather sufficient knowledge, which is crucial for determining an appropriate expandable initial state. Object information is selectively saved only when objects are within immediate reach (0.25m ahead). This selective saving facilitates the conversion of observations into a symbolic state and reduces the length of generated plans. The authors' findings suggest that the performance enhancements achieved by EPA over FILM and similar approaches primarily stem from its iterative planning approach. This approach enables the agent to recover from failure scenarios through flexible subgoal ordering, leading to improved performance. According to the authors, EPA achieved the second-highest ranking on the ALFRED leaderboard, closely following the Prompter method. The superiority of Prompter, however, is attributed to modifications in obstacle size and reachable distance rather than the use of prompts.

Strengths

1. The paper demonstrates the use of preconditions and effects through PDDL to improve the overall success of long-horizon tasks, especially unseen success rate. 2. The paper combines the use of symbolic planning using learned vision and language models and highlights how certain aspects of generalization can be achieved by abstraction.

Weaknesses

1. The current assumptions on semantic spatial graphs require random exploration for 500 steps to visit each location and form a node in the graph. The paper reports a drop in performance if this initial observation phase is ignored. This approach has two major limitations: (1) it dramatically increases the timestep overhead for the proposed agent as compared to the existing works. (2) it assumes a static unchanging environment after mapping and will likely fail in realistic environments with dynamic obstacles. Given the existing visual-inertial SLAM approaches, as noted by the authors, it seems that this issue can be mitigated. Some existing approaches for topological mapping [1] might also be relevant, and in turn, improve the path length weighted success rate (PLWSR). 1. Writing PDDL domain definitions and problems for each task is known to be a tedious coding task. Any errors in representing the available objects and actions would yield no plan. The current approach seems too close to reverse engineering the process of creating trajectories for the ALFRED task. While the authors report 150 hours for PDDL domain and problem definitions, a large chunk of the work involving object types and action predicates is already described in the ALFRED metadata. This does not give a reasonable perspective on how many hours would it take to scale and maintain this approach further, especially in the physical world. *[1] Chaplot, D.S., Salakhutdinov, R., Gupta, A. and Gupta, S. 2020. Neural Topological SLAM for Visual Navigation. In CVPR.*

Questions

1. Could you kindly provide some clarification regarding the semantic spatial graph? Specifically, what information does it contain and what is the average size of such a graph? The description in Appendix F2 suggests that the graph incorporates visual observations, segmentation, and depth. I would like to understand if this graph represents the visual scene "in front of" the agent. If so, I have a couple of related questions: 1. In case the agent is at the same location but facing a different direction, would the information on the graph be overridden? 2. If not, does each node in the graph contain a representation of the "360-degree visual observation" at that particular location? How are the "actions" represented as edges in this context? 2. I'm curious to know how the approach encodes common sense knowledge or visual knowledge (mentioned in Lines 57-58) as part of the domain definition in PDDL. 3. Could you please explain how the "exploration actions" are defined? In Appendix F3, Listing 3, it is mentioned that the agent cannot hold something and explore the environment. For example, can the agent actively search for the coffee after picking up the cup? I would appreciate some clarification on this. 4. I would like to understand how the possible predicates are listed in the PDDL. Additionally, how are the preconditions and effects identified for each predicate? Are they learned or inferred from visual observations, language goals, or the agent's interaction to gather information? Are these predicates hard-coded by a human? If so, I'm interested in understanding how this differs from the classical task planning setup. 5. It is not entirely clear how the generalization to new tasks is achieved, particularly when the language module is trained to output a task type out of seven tasks (as mentioned in lines 73-74). Could you please elaborate on the meaning of the statement in lines 308-309, "Our egocentric planning breaks a task into a set of goal states, autonomously generating an action sequence"?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Overall, it appears that the current approach is heavily focused on reverse engineering the trajectory creation process for the ALFRED task, utilizing learned vision and language models. While the method demonstrates a high success rate on the ALFRED benchmark for unseen scenarios, there is room for improvement in terms of clarity and the overall significance of the proposed approach. It remains uncertain how applicable the use of semantic spatial graphs or PDDL domain definitions would be beyond the ALFRED simulated benchmark, particularly when considering real-world physical environments.

Summary

This paper studies the problem of embodied tasks in which the agent needs to plan over long task horizons given natural language instruction. Current methods that use end-to-end training leads to entangled representation, which makes it hard to solve the task. On the other hand, planning methods such as PDDL can produce high-quality actions given a well-defined problem specification. To solve the task, the paper proposes a method that consists of two parts: (1) goal-oriented exploration that aims to gather missing information, and (2) a classical planning method that aims to produce a feasible action. The proposed method is evaluated on the embodied benchmark ALFRED. The method improves the prior SOTA performance by 8.3%, winning the ALFRED challenge at CVPR 2022 Embodied AI workshop. To be more precise, the proposed method consists of several parts: (1) a visual module for semantic segmentation and depth estimation of the scene, (2) an egocentric planner for planning given the information gather, (3) a semantic spatial graph for scene memorization. At the beginning of the task, the agent is provided with 500 steps to explore its surroundings. After that, the information is converted to a semantic spatial graph for the input of PDDL. Figure 1 shows the method. The algorithm inerates using the following steps: (1) find a path for reaching the goal, (2) if the goal is not reached, do exploration again Table 1 shows the main result of the paper. And the rest of the experiments section provides a detailed ablation study of the method.

Strengths

1. The writing of the paper is clear and easy to follow. For instance, in the introduction, I can easily understand the motivation of the method. 2. The proposed method is a winning approach at the CVPR Embodied AI workshop in 2022. This shows that the method has been thoroughly tested, and the result is convincing. 3. In the AI era, it is nice to see a classical planning method is robust in such tasks over a neural network-based method. 4. The related work is reasonable. It covers important papers such as Saycan 5. Overall, I think this is a good paper, and the method is elegant and promising.

Weaknesses

The proposed method could be very specific to the ALFRED task, in the sense that the method is solely optimized and engineered for the ALFRED task. For example, in some tasks, there is no abstraction of the action space, like OpenDrawer.

Questions

1. If I want to deploy such a method in the real world, what would be the challenge and the additional step to do this? 2. Following the previous questions, what is the Sim-to-real gap here? 3. Will the method be able to handle tasks such as you need to open the drawer to find the coffee mug in it, instead of target objects being visible to the agent? I will say this is more challenging in the POMDP setting than the setting in the paper. 4. What are some failure cases? Could you provide a couple of examples? 5. Is it possible to integrate such a method with semantic exploration (https://devendrachaplot.github.io/projects/semantic-exploration)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The paper has clearly stated the limitations of the paper, including the method cannot recover from irreversible failures, being sensitive to perception errors, and having no memory for belief tracking.

Summary

The authors propose a hybrid approach leveraging neural perception models and symbolic planners for egocentric planning and task completion in embodied environments. They demonstrate their approach in ALFRED benchmark winning the 2022 CVPR challenge. Their central idea is the use of symbolic planners, which are typically used in fully observable settings. To deal with partial observability, they perform iterative exploration with symbolic planning. Structurally, their implementation is similar to the previous SOTA on ALFRED — FILM. However, instead of using a semantic map as FILM does, they use a graph structure for object and location information storage. This graph is updated through exploration and then used by the downstream symbolic planner.

Strengths

- The authors’ hybrid approach leveraging symbolic planners is novel amongst the modular approaches for long-horizon EAI tasks, and can be promising for future research. - The authors won the CVPR 2022 EAI ALFRED challenge.

Weaknesses

- Poor clarity of exposition: I found the major aspects of authors’ technical approach poorly explained in the paper. This really limits reproducibility as well as use by the community IMO and consequently limits the value of the paper. - Spatial graph: Unclear how the graph is built and what does it look like. The authors say that the graph encodes location as the node key and visual observations as values. However, it is unclear what is the co-ordinate system for location (consequently would we get the same graph for the same environment and task but different agent start position?), what happens when the agent visits the location more than once, and lastly, how is it used for the low-level policies that have to navigate given there is no map. - Object-centric POMDP: The authors briefly mention the Cost POMDP but do not appropriately define or explain it at all in the main paper. Most definitions are pushed to supplementary. While it is okay to push nitty-gritty details to supplementary, without properly framing the problem and then grounding their approach in the framework, I am not sure if I can buy the authors’ claim that their approach is theoretically sound (L121). - Exploration: Unclear how exploration is handled. The only description is in the intro (L52-56). - The algorithm is not commented and contains symbols which are never defined or explained, making it difficult to understand what did the authors actually implement and why: - Symbol Ge is never defined - Unclear how cost c of action is computed and used in function $M^{PD}$. - The authors say that the PDDL domain and environment definitions of actions are misaligned, which make sense (L149). However they never explain how this alignment is obtained in their approach. - Furthermore, given the PDDL and planning terminologies might not be accessible/known to everyone, I encourage the authors to be more careful and clear in their descriptions. For instance, what exactly are anchor object types and why do we need them? Also unclear what other anchor object types can exist in EAI settings. - I didn’t get much out of Sec.4 and 5 despite the fact that I work with symbolic planning and EAI and even when there is so much to explain about the approach as mentioned above. I recommend rewriting entire Sec.4 and 5. - Generalization: Authors claim and attempt to show generalization in Sec.8.4. However, this section had no details whatsoever about the new tasks they use nor a reference to supplementary. I managed to find some details in supplementary. However, I am sure a general reader will be lost here. Furthermore, the authors claim that they do better on the new tasks and achieve zero-shot success 82% times, better than other methods but do not show any such comparison in their work (tab.6 supplementary). Given this is one of the main claims of advantage of their approach, I’d like to see comparison with neural and template-based approachers, and also with the prompter method. With the above (also see limitations section below), IMO the paper is not yet at par for publication at Neurips. In its current state, it can be a great workshop paper but requires major rewriting and additional comparisons e.g., for generalization claims otherwise.

Questions

- Table1: main results should have bolded numbers and indications on which metrics need to be higher/lower for better task performance (e.g., with $\uparrow$ and $\downarrow$) to improve interpretability of results. Similarly sorting rows in Tab2 based on performance in unseen might be useful. - L166: “This setup promotes more robust action sequences and generalization beyond the seven ALFRED-defined tasks” — not sure how? - Results: Given that the LGS-RPA also comes close in terms of GC and SR (unseen) to the authors approach, I encourage the authors to discuss LGS-RPA in the result section as well. - Tab:3 Unclear why the method only achieves ~60-70% GC with all ground truth available. Is this the upper bound? Why is the upper bound not 100% given that the env. is deterministic and we are using a symbolic planner? - Is the planning performed from scratch every time? I imagine that the state of the world and thus the graph have incremental changes so wondering if the authors do anything smart to reduce the planning time at each iteration.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

- The authors do not touch upon the issue of obtaining PDDL domain in the first place, which is core to their approach. Unclear how this would scale for real world applications/agents and how errors in domain file can impact their approach. Lastly, they mention that they can handle additional constraints e.g., energy during planning. However, in practice, I’ve found planning time to blow up when using durative and cost constraints. Unclear again how authors approach would scale for more complex problems. Also, on that note, I’d like to see planner time numbers for the tasks perhaps in the supplementary. - Similarly, the authors say that they are more robust to perception errors however they say they do so using hand-crafted policies for replanning (Sec.8.3). Unclear if the robustness is because of these handcrafted policies or because of their iterative planning approach combining exploration and symbolic planning. - The section on broader impact and societal impact is missing. I'd encourage the authors to think about real world applications that their work might enable for this section.

Formulas format for revisions in sections 4 and 5

We realized that the format of the formulas did not work properly in the response. This revised format may clarify the visualization: - In the revised section 4, we define an *Environment* as a tuple: $\mathcal{E} = \langle A_\mathcal{E}, \mathcal{T_\mathcal{E}}, \mathcal{V_\mathcal{E}}, \mathsf{reset}, \mathsf{step} \rangle$, with corresponding definitions for each term. - A *Task* is defined as a tuple: $T_\mathcal{E} = \langle I_\mathcal{E}, G_\mathcal{E} \rangle$, setting the initial state and the goal through the $\mathsf{reset}$ function. - A new task can be considered a new initial state in the fixed Object-oriented POMDP, including hidden variables to express goals and user preferences. - In the revised section 5, Algorithm 1 receives as input: - Environment $\mathcal{E}$ - Planning domain $\mathcal{PD}$, provided by the user - Anchor types and corresponding exploration actions, provided by the user - Agent’s mental state initialization function $\mathcal{M}^I_{\mathcal{PD}}$, mapping a task and initial perception into a belief state - Agent’s mental state update function $\mathcal{M}^U_{\mathcal{PD}}$, updating the agent's state and returning new planning objects and a modified initial state for the replanning loop - Algorithm 1 returns a successful trace or Failure if exploration failed and no plan to the symbolic goal was achieved. - For ALFRED, the mental state initialization and update create and refine the spatial semantic map. - Our contribution is a method for solving a series of tasks $T_\mathcal{E}$ for the same environment $\mathcal{E}$. - While the ALFRED challenge includes only seven task templates, our method generalizes to other tasks within the same environment. So, our method is more appealing when users require more flexible generalization.

PDDL symbolic models

We thank the reviewers for their insightful comments and questions. We hope we have addressed them satisfactorily. If there are any remaining concerns, please do not hesitate to let us know. Regarding the authoring of the PDDL symbolic models, we understand that obtaining a symbolic model might be considered non-trivial. It's essential to differentiate the convenience of representing actions as PDDL symbolic models from the process of obtaining them. We included a reference to [AKFM22], which explores learning to plan given pixel states and unlabeled actions. The paper utilizes a sequence of variational auto-encoders for learning an action model, demonstrating that symbolic composable action models contribute to high generalization. For our context, learning a PDDL is relatively simpler since we assume that the environment includes object types and actions. We've added a citation to the review [AFP+18], which focuses on learning planning action models. In our submission, we assume that the user provides a symbolic planning model with full observability. For ALFRED, we further assume that actions are deterministic. This simplifies the problem into classical planning, enabling the use of various planners and tools. - Symbolic planners are available in ([Planutils](https://github.com/AI-Planning/planutils)). [MPSK22] - There is a [VSCode extension for PDDL](https://marketplace.visualstudio.com/items?itemName=jan-dolejsi.pddl) that has been actively developed since 2017. Moreover, recent advancements in pre-trained large language models offer opportunities for PDDL modelling and for translating natural language tasks into symbolic goals. An article under review titled “Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning” explores this further. See this [interaction with ChatGPT](https://chat.openai.com/share/c0cd35b5-dfdb-4320-972d-a9188d77f631) for examples. - [AKFM22] Asai _et al_. Classical planning in deep latent space. J. Artif. Intell. Res., 74:1599–1686, 2022. - [AFP+18] Arora _et al_._A review of learning planning action models_. The Knowledge Engineering Review, 33(e20):1–25, 2018. - [MPSK22] Muise _et al_. PLANUTILS: Bringing Planning to the Masses. Demo track of ICAPS 2022. [PDF](https://icaps22.icaps-conference.org/demos/ICAPS_2022_paper_377.pdf)

Thank you for your answers

In first place, I would like to thank the authors for their responses. I would also like to clarify the point of one of my questions. Q1. I was actually interested in a high-level description of the management of the information graph/mental state in each agent. For example, how would the state of objects that have undergone changes without the agent's interaction (e.g. caused by the environment, or by an external agent) be updated?

Thanks for detailed rebuttal

Thanks to the authors for individual responses to the reviewers and the general response. These span writing clarifications and fixes, generalization claims, sim-to-real gap, PDDL models and domain definition, assumption of deterministic state transitions, and specificity of contribution to ALFRED benchmark. The committee shall discuss further and take these into consideration.

Thank you for the detailed rebuttal

Thanks for providing various clarifications, a graphic for the graph, and updating symbols and descriptions. I am a little concerned that many technical descriptions in the paper have been changed but not reviewed fully (as in initial reviews), but I'll let the ACs decide if it would be okay to accept the paper in such a case. Either way, I'll raise my rating to "weak/borderline accept" -- I will not argue for the paper but I won't push back if other reviewers are supporting the paper/there is a champion for the paper. - It would be good to provide clear distinction on LGS-RPA and why a comparison isn't possible in the main paper. Same for the tasks used for generalization, to explain why other baselines are not possible. - I am also concerned about the planning time numbers for real-world applications, perhaps the authors can add that in their discussions/limitation section. - Also glad to see more clarity on PDDL assumptions and prior work on learning PDDL. Hoping authors can do the same in the main paper.