Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity

Summary

The paper explores introspective planning to enhance robotic task execution using large language models (LLMs). The authors introduce a method for LLMs to form uncertainty-aware plans without fine-tuning while addressing hallucination and task ambiguity. Their approach integrates introspective planning with conformal prediction.

Strengths

The use of introspective planning with LLMs. The write-up is easy to follow. Sufficient comparison and ablation experiments.

Weaknesses

NA

Questions

NA

Rating

7

Confidence

1

Soundness

3

Presentation

4

Contribution

4

Limitations

The authors addressed the limitations of their proposed work.

Summary

This paper introduces a method that uses introspective planning to guide LLMs in forming uncertainty and ambiguity-aware task plans. The proposed method derives and quantifies the inference uncertainty of LLMs to enhance task planning and conformal prediction. Additionally, a new dataset on safe mobile manipulation is created as part of this work.

Strengths

+ Leveraging LLMs to improve robot task planning performance is promising. + The concept of introspection is interesting. + The created dataset on Safe Mobile Manipulation benchmark could benefit the robotics community.

Weaknesses

- The term "introspection" or "introspective" is not scientifically defined. Additionally, does introspective refer to a robotic agent, the LLM, or the proposed approach? - The novelty of the paper is unclear, especially compared to [31]. Line 63 states "The fundamental aim of introspective planning is to guide LLMs automatically to reason about task uncertainty and assess the feasibility of candidate actions." [31] also reasons about uncertainty and ambiguity through MCQA. Is [31] introspective? - The paragraph in Line 135 indicates the significant enhancement is from the introspective planning rationale k_i, and Line 48 indicates that introspective planning provides a tighter statistical guarantee. However, no theoretical or mathematical proofs are provided. - How does the proposed approach enable the new capability of modeling and addressing task safety? The method of handling safety seems identical to addressing ambiguity through MCQA. - What robotics simulations or physical robots are used to create the safe mobile manipulator dataset?

Questions

- Refer to the comments in the Weaknesses section.

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

No negative societal impact of the work is perceived.

Summary

The paper tackles the uncertainty quantification problem for task planning with LLMs. Specifically, the paper proposes to first construct a knowledge base using LLM, that contains human-in-the-loop correction and LLM summarization / reflection. Then this knowledge base is used during inference to provide relevant examples in a retrieval-augmented generation (RAG) fashion. Finally, this is combined with conformal prediction to either predict the next action step or ask for clarification due to high uncertainty. Evaluations are performed on a set of text-based task planning datasets and demonstrate improved performance compared to various baselines on multiple metrics.

Strengths

- Uncertainty quantification is an important topic in the context of task planning with LLMs, and the proposed method shows improvement on most of the metrics compared to prior works (and I also find the metrics to be reasonably constructed) - The paper overall is well-written and easy to follow. Figures are intuitive and helpful for understanding the core contributions.

Weaknesses

- Despite the improvement, the significance of the contribution is slightly unclear when compared to prior work “KnowNo” - it seems that the only difference is that there is an additional “chain-of-thought style LLM summarization” step for the knowledge base and the calibration dataset. Although it is intuitive that additional improvement can usually be gained by chain-of-thought, it remains unclear if the gain is marginal when there is better underlying LLM.

Questions

See "weaknesses" section above.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are described in the paper.

Summary

This paper presents "introspective planning" as a method to enhance the reliability and safety of robotic task planning using large language models. The proposed method uses introspective reasoning to address uncertainty through a knowledge base consisting of sets of tasks, observations, and candidate plans, along with a rationale for plans to better align with user intent. Additionally, the authors introduce the Safe Mobile Manipulation benchmark with safety-critical scenarios and new evaluation metrics.

Strengths

1. The ability to generate uncertainty over robot tasks, especially in cases of unsafe operations, is particularly important. 2. The paper is well motivated in aligning LLM-generated tasks with the true user intent, given possibly ambiguous inputs. 3. The knowledge base includes a rationale for each task which expands on the KnowNo framework for generating confidence scores. 4. The new additional safety benchmark adds important safety scenarios to existing datasets to better evaluate planning under uncertainty systems. 5. The method is described well, and the availability of the code gives the opportunity for the broader community to build on this work.

Weaknesses

1. The core components of the work are well established. In the proposed direct method, the incremental improvements to existing work are primarily the introspective approach where the LLM is used to generate a rationale for the plan. The conformal prediction method additionally uses a knowledge base to produce statistical guarantees about the prediction which is very important for safe operation. However, the results show significant performance gaps between the direct and conformal prediction. Given this, it would be good to provide a robust analysis of this tradeoff in a general setting. 2. The paper focuses primarily on manipulation tasks that involve somewhat ambiguous items in a kitchen setting such as disambiguating between two sodas in the scene or having the knowledge that a plastic object shouldn’t go into an oven. It would be helpful to include other domains to better show the generalizability of the method. 3. The knowledge base is a key component of the conformal prediction. The authors compare different sizes of knowledge bases, but little is given towards how these should be constructed for a given task and domain. More details on the variations that a user needs to generate should be given. It seems the user needs to be well aware of the failure mode of the tasks to generate an appropriate knowledge base. 4. There may be bias in the user-generated knowledge base, especially for multiple users. It would be good to show how these affect the performance of the system. Perhaps evaluating a system with multiple users without sharing the intended goals would show how well the system aligns across multiple users and a single knowledge base.

Questions

1. It wasn’t initially clear to me whether the knowledge base contains the explanation. Are these stored after construction - they are not in the data files in the code repo. 2. The experimental exploration of the knowledge-base size contained in the appendix is appreciated. Is there a reason that the other metrics were not included in this evaluation? Do you have insights into why the performance drops with the largest knowledge base and what was the variation in tasks contained in the knowledge base? 3. What is the result when the user provides an ambiguous answer when the system seeks clarification?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

1. As the authors point out, one limitation is that the conformal prediction seems to perform significantly worse than direct prediction. Particularly interesting is that the Unsafe Rate is worse for conformal prediction than direct, which might raise the question as to how much the uncertainty measurement is contributing to safer outcomes. 2. The method is evaluated on a limited diverse set of tasks in a kitchen manipulation setting. 3. The authors make a good observation about multi-label prediction and more investigation is needed to understand the limits around truly ambiguous tasks.

Summary

This paper proposes a new method of using LLM to do task planning. The key innovation is a retrieval-augmented generation (RAG) where the LLM retrieves few-shot introspective reasoning examples from a knowledge base that contains examples with supervised labels. The authors integrate the RAG into LLM to either (1) make it directly predict the best plan, and (2) make it use conformal prediction to predict the best plan. Results have shown that the proposed method, LLM + introspective reasoning + direct prediction outperformed all baselines. And LLM + introspective reasoning + conformal prediction outperformed all baselines with conformal predictions. However, LLM + introspective reasoning + direct prediction outperformed LLM + introspective reasoning + conformal prediction, which is an open question for future work.

Strengths

- The method is sound where the main message is very simple - introspective reasoning based on a dataset with ground truth labels is helpful for LLM planning. - The writing is very clear. - This paper also proposes new evaluation metrics, which more comprehensively measures planners' performance. - The experiment is rich with 3 problem domains with various baseline methods, including the non-conformal-prediction-based and the conformal-prediction-based. - The result has shown the benefit of incorporating introspective reasoning.

Weaknesses

Besides the limitation discussed in the end of the paper, there are two other potential limitations: - The proposed method relies on a knowledge base with human supervised labels, which may require significant human efforts to construct. It might be useful to discuss the cost of such knowledge base. - The usage of both introspective planning and conformal prediction might increase the computation load for inference. It might be useful to discuss the increased computation.

Questions

- In Eq.2, why choosing (N+1)(1-ε)/N? - Line 135-141 highlights the difference of the proposed work vs [31]. Just to make sure I understand it correctly, the key difference is incorporating `k` in Eq.3?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations are discussed in the end of the paper.

keeping my score

Thank you for the detailed explanation. After reading other reviews, it seems that one main weakness is whether integrating introspective reasoning is sufficiently novel. As I am not familiar with the literature in LLM for planning, I will defer to other reviewers about evaluating the contribution. For now, I am inclined to keep the score.

Response

Thank you for the response -- it has addressed my questions, and I'm inclined to keep my rating.

Response

Thank you providing additional clarity to my questions and comments. Also, thank you for providing the additional plot on the size of the knowledge bag. This clarifies my understanding of the sytem and confidence in my rating.