Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation

Summary

This paper proposes Demand-driven Navigation (DDN), which leverages the user’s demand as the task instruction and prompts the agent to find an object which matches the specified demand. This paper also proposes a method of first acquiring textual attribute features of objects by extracting common sense knowledge from a large language model. These textual attribute features are subsequently aligned with visual attribute features using Contrastive Language-Image Pre-training (CLIP). They experimented on AI2Thor with the ProcThor dataset and demonstrated that the visual attribute features improve the agent’s navigation performance and outperform the baseline methods.

Strengths

The paper flows nicely with good motivation of the novel task and its challenges clearly stated. The proposed method makes sense and comparison is good.

Weaknesses

1. L40, the second condition is a bit questionable. I don’t think it is a necessary requirement of only searching objects that are in the scene. It is probably the case in current benchmark setup, however it is not a constraint for the research task. Subsequently, it might also impact the definition of “navigation failure” at L48. Asking the robot to search for a non-existent object in the scene and the robot could not find it. This should not be defined as a failure from my point of view. Rather, if you ask a robot to look for some non-existent object and it reports a finding, this should be counted as a failure as it is a clear false positive detection. Nevertheless, the new task DDN does not necessarily remove ‘requirement’ 2) as you might anyway asks for a demand that no object in the scene can satisfy. 2. From L171, my impression is that the problem is constrained by only one demand d at a time. In reality, one could have multiple demands to further constrain what object they might want to search. For example, something “quench thirst” and “contains caffeine”. 3. It will be more convincing with more benchmark datasets. In addition to AI2Thor, there is Habitat Challenge on ObjectNav which can be exploited, plus it is based on 3D scan of real scenes.

Questions

1) Can we handle a search composed with multiple demands, with the current method? 2) Why other benchmark dataset are not used for evaluation? is there any justification?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discussed both limitations and the societal impact.

Summary

This paper presents a new visual navigation setting, where the goal is not specified by objects or images but described by a sentence. The sentence encodes the essential information to search for specific objects during navigation. Different from VLN, the task is able to analyze the demand within each sentence rather than step-by-step language guidance. Common sense and knowledge will be explicitly extracted from a large language model. To align the visual and attribute features, CLIP is employed. Overall, the motivation of this work is reasonable and interesting. # Post rebuttal The authors have addressed most of my concerns. If the authors can provide some visualization results, that would be more convincing. Therefore I changed my score to accept.

Strengths

The motivation of this work is interesting. Providing a demand description to an agent would enable the agent to search for not only one object in order to complete the specified goals. The authors also provide semi-automatically generated data for this new task. This would be complementary to the existing VLN or VN tasks.

Weaknesses

The introduction part is overly lengthy. The authors exert three pages to describe the motivations of this work, making reading quite tedious. I highly suggest the authors could trim the introduction part a bit. The natural questions come to this task is whether the proposed method can complete object-goal navigation after training? For example, after the network is trained based on demand driven sentences, whether it can be used as a object-goal navigation agent? The common knowledge or sense is pre-defined. In the illustrative figures, it seems a demand may correspond to three different objects. Whether this would restrict the options? In L189, the WG mappings are different depending on the environment. I am not sure whether this implies that these WG mappings need to be specified manually. If so, this may contradict with the original motivation of this work, where humans may not know the environment in advance.

Questions

My questions mainly focus on two tasks: In this comparisons, the results of some baseline methods are significantly lower than the results reported by their original papers. Therefore, I am wondering how the authors adapt their methods to this setting? A few works leverage CLIP for VLN or VN tasks. CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments These works can be adopted for comparison. As the searched objects have strong association with the demand, what if some demands cannot be processed properly? For example, I am thirsty but I cannot drink cold. Simply providing demands and their corresponding objects would lead to overfitting.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

There is no negative societal impact.

Summary

This paper introduces a new task called Demand-Driven Navigation (DDN) that, unlike previous Visual Object Navigation (VON) tasks that evaluate the ability of an agent to find a specific object in an unknown environment, considers fulfilling the demand of a human. This new task is motivated by the lack of real-world grounding of current VON tasks that either require an agent to find an instance of an object category from a pre-defined fixed vocabulary or a language-specified object in an open-vocabulary fashion. However, in a real environment, a specific object might not be present or, if thinking from the point of view of fulfilling a human’s demand, other objects might be equally as good as the targeted one. As a result, authors suggest querying an autonomous agent with a language demand instruction. This allows one to be more flexible with respect to environments but also requires common sense knowledge, an understanding of how objects can be used, and where they are likely to be located in a scene. The paper evaluates several baselines inspired by previous literature to show the introduced task is hard and cannot be solved with currently known methods. A new approach to solving the DDN task is thus introduced in this paper. The main goal is to learn a mapping between human demands and attributes of objects that can fulfill them. At training time, a GPT-3 model is thus used to generate a series of demands and objects that can meet each of them. Each demand is encoded by a BERT model and each object is encoded by a CLIP text encoder. For each demand, a demand-object vector is created by concatenating the demand representation and an object representation. An attribute module takes the demand-object vector as input and is trained with contrastive learning to extract representations that are as close as possible for different pairs sharing the same demand. This attribute module is then used in the final policy that is composed of a Transformer model and is trained with imitation learning. The paper shows this new method outperforms other baselines.

Strengths

1. The introduced task is very interesting: querying an agent with a human demand seems more aligned with what is needed when deploying a robot in a real environment to help humans. Authors have properly motivated the caveats of current VON tasks in the literature. 2. The extraction of object attributes seems very relevant and well-motivated. Learning attributes is indeed a way to guide the learning of common sense knowledge and experiments showcase the gain in performance it allows to reach. 3. Authors propose diverse baselines to evaluate the performance of current language models in robotics/navigation tasks.

Weaknesses

1. [Major] The introduced method is composed of many different modules and pre-trained models (GPT-3, BERT, CLIP text encoder, CLIP vision encoder, Attribute Module, DETR, Image Encoder, Policy Transformer, Visual Grounding model). It is not clear what parts are the most important and whether the overall method could maintain the same performance without some of these building blocks. Additional ablation studies would be very interesting. 2. [Major] Most baselines except *Ours* showcase very low and close performance. However, section 6.3 discussing the experimental results is quite long and detailed. I am not convinced authors can draw as many conclusions as they do when we consider how close all baselines are in terms of average performance (even more when considering the standard deviation). 3. [Major] This remark is very related to the previous one. All concurrent baselines reach very low performance. Could it be that these methods were not trained to convergence or would simply require much more training? The authors mention 1.8M training frames in the paper, which seems rather small compared with the number of frames required to train baselines in other navigation tasks (generally closer to 100M training frames). 4. [Minor] When presenting baselines in section 6.2, a lot of information is missing. The paper refers to details given in the supplementary material. When reading this supplementary information, we can understand the introduced baselines. I would suggest including this information in the main paper directly.

Questions

Four questions (1.-4.) were already asked in the "Weaknesses" section. I would like authors to address these concerns. I also add two additional questions (5., 6.), that do not appear to me as paper weaknesses but rather required clarifications: 1.-4. See "Weaknesses" section. 5. [Major] When describing the *Random* baseline, authors say it is about randomly selecting an action in the action space. I thus do not understand the difference between *Random-object* and *Random-demand*. Further clarifications are needed. 6. [Major] Performance for *GPT-3+Prompt** and *MiniGPT-4* in Table 1 is always the same independently from the scene (seen/unseen) and instructions (seen/unseen). This should either be corrected or explained.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The paper mentions limitations regarding the drawn conclusions about the performance of current language models on the DDN tasks. Authors explain they did not have access to the code for recent methods such as GPT-4 with visual inputs or PaLM-E, and thus were not able to evaluate these methods. It is a good thing to mention this limitation, but their experiments were already conducted with many recent models, which might still allow them to draw relevant conclusions (see “Weaknesses” section for remarks about the drawn conclusions, which are however orthogonal to the mentioned limitations in this section).

Summary

This paper proposes a Demand-driven Navigation (DDN) problem to leverages the user’s demand as the task instruction and prompts the agent to find an object which matches the specified demand. Then the authors proposed a method by learning demand-conditioned object attribute features from LLMs and align them to visual navigation via CLIP. The experiment shows the efficiency of the proposed method. However, I have some concerns about this paper. My detailed comments are as follows.

Strengths

1. This paper proposes a novel Demand-Driven Navigation task to explore the navigation with only the user’s demand as the task instruction. This task is practical and worth more research, especially with the development of open-vocabulary foundation models. 2. The proposed attribute module is interesting and helpful for extracting attributes of objects. 3. This paper provides a method to tackle the DDN task by extracting common sense from LLMs to learn textual attribute features and uses CLIP to align the textual and visual attribute features. The results obviously outperform the baselines.

Weaknesses

1. One important baseline is missing. The agent could explore the environment using a heuristic algorithm like FBE. At each time step, the agent detects all objects in observation and ask LLM whether these objects can satisfy the human demand. 2. What are the differences between common sense knowledge and human preferences mentioned in the paper? 3. It is not clear why the results of random-object are different from random-demand. Are they both execute random actions? More explanations are needed. 4. In Table1, the ZSON-demand performs better the ZSON-object. Does it indicate that CLIP performs better in understanding high-level abstract demand compared to concrete objects? This result seems to conflict with the claim that “CLIP does not perform well on alignment between instructions and objects” in Line 335. 5. Some related works that try to solve open-vocabulary navigation[1,2] or scene understanding[3] are missed. It would be better to add and discuss them in the related work part for the sake of completeness. [1] Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation, NeurIPS 2022. [2] Visual Language Maps for Robot Navigation, ICRA 2023. [3] LERF: Language Embedded Radiance Fields, ArXiv 2023.

Questions

My main concerns are the lack of an important baseline and the analysis of the experimental results.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations

Summary

This paper proposes the novel task of demand-driven navigation, where a robot must navigate towards a goal object that satisfies the human user's demand (e.g., for the demand "I am thirsty", the robot has to find water/juice/tea, etc.) Citing limitations of navigation methods used for other object-goal navigation variants, the paper also proposes a novel architecture that relies on extracting attribute features conditioned on the demand text features and object features for visible objects. The attribute features extracted are indicative of what physical / semantic properties are fulfilled by a given object to satisfy the demand (e.g., the object "water bottle" has the property to "quench thirst"). These features are learned using a contrastive learning objective on prior-knowledge extracted from a large-language model (GPT-3). Experiments on the ProcThor dataset demonstrate the difficulty of the demand-driven navigation task and the superiority of the proposed method over prior navigation methods.

Strengths

* The idea of demand-driven navigation is interesting and novel. While the goal specification is natural language (similar to prior work), it focuses on a very different concept of "demand" where multiple functionally equivalent objects can satisfy a given demand. * The problem statement is well motivated and the paper writing clarity is largely good (see weaknesses for some questions). * The supplementary material provides necessary implementation details and details about the dataset for reproducibility. * The experimental design is good. Recent baselines for open and closed vocabulary navigation are considered. Multiple experimental trials have been performed to show statistical significance. The proposed method significantly outperforms the baselines.

Weaknesses

# Post-rebuttal update I thank the authors for addressing my questions and concerns in the rebuttal. It is clear to me that this paper presents a valuable new direction in this space of navigation tasks and the experiments are sufficiently strong to recommend acceptance. The motivation can be clarified further and the rebuttal responses need to be reflected in the final paper. With the understanding that this will be done, I am increasing my rating to accept (7). --------------------------------------------------------------- ## Task motivation good, but practical implications are not clear I liked the task itself, but a key question that concerns me is *"how often does an object demanded not be present in the scene, and therefore, a functionally equivalent object was needed to satisfy the demand?"*. That is, how often do demands for objects become infeasible because the object was missing in the scene? E.g., if the demand is "Get me a water bottle because I am thirsty", I would expect most scenes would contain water bottles. The task definition avoids addressing this issue by having a generic demand itself as the input. Of course, it is possible that there are other objects that can be functionally equivalent **in addition** to the object demanded, but that's not the scenario motivated in the introduction. Additionally, as a user, if I request a water bottle, that's exactly what I'd want unless it is nowhere to be found. So the proposed task makes more sense in the absence of the primary object. ## Dataset is underwhelming L63 - 64 - "mapping between demands and objects is many-to-many" ---This is only partly true. Based on statistics in Figure 4 supp., only 2.3 objects, on average, correspond to a given instruction. Calling this "many" is underwhelming. The other direction is still true though, i.e., there are many instructions satisfied by a given object (Figure 5 supp). ## Task definition fails to consider whether an object instance satisfies the demand L89 - "Both a bottle of water and a cup of tea" can "quench thirst" --- this is only true if the bottle has water and cup is filled with tea. Does the dataset / task differentiate between bottle/cup instances that contain liquids vs. those that are empty? ## Approach clarifications needed * L175 - "only take RGB images as sensor inputs" --- is the GPS+compass information also included here or does the model learn to localize on its own? * L185 - why is the time step limit only 100? That seems very short for navigation in large environments. * L257 - why is the attribute module a transformer (e.g., why not just an MLP)? Is self-attention across demand-object features needed? * L281 - why use only imitation learning and not reinforcement learning? ## Experiment clarifications needed * Table 1 - why is the performance on seen scene, seen instruction so low? I'd expect close to 100% success due to overfitting. * L306 - how are the object categories derived from demand inputs for the *-object methods? * L317 - why are there variants of "Random" if the policy only selects actions randomly? * L336 - 338 - "likely to be due to the fact that ... have a high likelihood not to be present ... meaningless search" --- can we empirically quantify this? How often is it the case that objects predicted by GPT-3 are missing in the scene? * L371 - 373 - "surpasses all baselines ... CLIP-visual features helps ..." - it seems to me an unfair advantage to use CLIP visual features only for the proposed method and not the baselines. How do baselines perform when equipped with CLIP features?

Questions

I'd appreciate it if the authors can clarify the questions raised in weaknesses. It will help me arrive at my final decision.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, the limitations are discussed.

Discussion with authors

Dear Reviewers, Please check the rebuttal and start a discussion with authors if you need any additional information to make your final decision. The discussions should be completed by Aug 16. Thanks, AC

Some clarifications on Q6 L306

We apologize that we misunderstood your question on the Q6 L306 at first. Here are some clarifications. Methods with the -object suffix are trained in **visual object navigation** and tested in **visual object navigation**. Instead of deriving from demand inputs, we directly inform the robot of the object category that the robot needs to find. We apologize for putting them together with the demand-driven navigation results that caused some misunderstanding. We will make a separate table for methods with -object suffix later. Methods with the -GPT suffix were used with models trained in **visual object navigation** and tested in **demand-driven navigation**. We get an target object category that can satisfy the demand instruction by asking GPT-3. Then we inform the robot of the target object category given by GPT-3. We will also explain the different suffixes in more detail in the main paper.

Thanks for the reply

I thank authors for the response, and I would sincerely suggest to add these clarifications in the paper. As I mentioned in the original comments, the new task makes sense and the method is convincing with the supported comparison. I'd appreciate a good clarity in the statement in terms of the assumptions and limitations.

More explanations about baselines' low performance

Thank you so much for reading the following. We explain the question in Q5 in more detail, especially in response to why the results of some baseline methods are significantly lower than the results reported by their original papers. **Q8**: In this comparisons, the results of some baseline methods are significantly lower than the results reported by their original papers. A8: There are several factors contributing to this observation. (1) It's important to highlight that the scope of object categories within the DDN task has expanded significantly, encompassing a total of 109 categories. In contrast, the original VON papers focused on a narrower range of objects: 22 categories in the case of VTN, and 6-21 categories for ZSON. This broader object category coverage inherently introduces greater complexity. (2) The DDN task involves the utilization of natural language instructions, resulting in a considerably wider description space than the VON task. This expanded description space inherently escalates the level of task complexity. (3) the VON methods do not take into account the many-to-many object-instruction mapping phenomenon present in the DDN task, resulting in their lack of reasoning about the combination of instructions and scenes. These are the reason why the VON methods perform lower on the DDN task than reported in their original VON paper. Our method uses GPT-3 to generate numerous language-grounding mappings for learning demand-conditioned object attribute features. By leveraging CLIP's capability to align visual and textual information, our method integrates instruction and scene details. We hope to hear back if you have further questions!