GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance

Summary

The paper is studying the problem of zero-shot object goal navigation. To solve the problems in traditional approaches, the authors propose a novel method, named Geometric and Affordance Maps (GAMap), which incorporates both geometric and affordance attributes as navigation guidance. A multi-scale scoring approach is proposed to cope with objects at different scales. Experiments on HM3D and Gibson datasets show the effectiveness of the proposed method for the metrics of success rates and success weighted by path length.

Strengths

1. The paper is studying the problem of zero-shot object goal navigation, which I think is an important problem in object-goal navigation. 2. The proposed method incorporating both geometric and affordance attributes is sensible to me. 3. There are extensive analysis in the experiment section of the paper, which provides several insights.

Weaknesses

1. Although the authors have demonstrated the difference of their work compared with previous ones, I am still not convinced about the key novelty of the work. In terms of organisation of the paper, it would be better if the authors could provide a table to better compare the difference of their work with existing works. 2. There seems no real world experiments. I am not fully convinced about the effectiveness of the proposed method unless results of real robot experiments are provided. 3. The authors have not removed the checklist instruction blocks. 4. In Line 214, there is an issue with the citations. The paper needs further proofread.

Questions

It would be good if the authors could address my concerns in the weakness part. I will make my final decision after reading the authors' rebuttal and other reviewers' comments.

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have discussed about limitation in appendix.

Summary

The paper presents an approach for the task of zero-shot object goal navigation. The approach first generates a geometric and affordance description of the goal object. This description is then matched with the visual embedding of the RGB frame (at multiple scales) in the CLIP embedding space to compute a similarity score. They project this score on an orthographic 2D map. During navigation, the agent explores the frontiers/unexplored regions with the highest score. They demonstrate the efficacy of their method on HM3D and Gibson object-goal benchmark.

Strengths

- The authors present excellent ablations in the paper, justifying each design decision with ablations. They show ablations for using affordance and geometric attributes, multiple scaling levels, different visual encoders, and even different methods for calculating and merging GA scores! - The results (success-rate) on HM3D and Gibson object-goal navigation benchmark are better than the next best method (VLFM) (1.1% on ML3D, and 1.7% on Gibson).

Weaknesses

- The idea of using attributes and longer descriptions for objects instead of just object label has been well studied [A, B] and I would have expected the authors to make connections to this popular line of work on effectively utilising CLIP representations. - I also thought that the 23.7% increase in SPL, and 37.0% in SPL claim is quite misleading specially because the only difference between them and VLFM is that VLFM uses a learned policy for navigation. It has been well established that using FMM for navigation works well enough for indoor navigation [A] Visual Classification via Description from Large Language Models, ICLR 2023. [B] LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions, 2023 [C] Navigating to Objects in the Real World, 2022.

Questions

- The authors argue that comparison to VLFM is unfair because they use a trained policy compared to FMM to achieve high SPL on HM3D. Even then, they obtain better SPL on Gibson. Can the authors comment on why a trained policy will help HM3D, but not Gibson? - I didn't fully understand Figure 3. What does decrease in success rate mean? Is it relative or absolute decrease? Is it percentage or absolute points?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors don't discuss any limitation of their work in the manuscript. - Maybe the authors should add a small section discussing potential limitations such as lack of real-robot experiments in the paper. - assumptions about perfect odometry / localization to build a map. Not a limitation, but a suggestion to improve the manuscript: - I thought that the writing could be clearer. Some of the sentences were overly complicated. For example Line 42-43 - "proposed method initiates with the LLM inferring the object’s geometric part attributes and potential affordances attributes, delving into a detailed understanding of the object’s physical structure and functional properties". The paper will benefit from more succinct and clearer writing.

Summary

The authors propose an algorithm for the object-goal navigation/exploration problem by using a 2D navigation map containing geometric and affordance scores for the target object. Geometric and affordance features for the target object are extracted by asking an LLM to list them for the object, and these words are then used with a multi-scale CLIP setup to get the resulting "GA" scores that are projected into a 2D map and accumulated over time.

Strengths

* The object navigation problem is important for robotics and embodied AI applications * The paper is mostly well-written and the experiments have ablations * The approach appears novel if somewhat narrow.

Weaknesses

* Narrowness of contribution: Projecting CLIP-like open-vocab scores into a 2D navigation map has been done in earlier work like VLFM [17]. The novelty here appears to be that you ask an LLM for multiple (geometric, affordance) properties of the object to query CLIP with. * Motivation of baselines: The results only seem about 1% better than the recent VLFM paper, which you dismiss because it contains "locomotion" training. However, no mention is made how your robot does motion planning. Since your results measure path length, you have to use something to get to a high-scoring frontier without colliding. The digitized environments are also just referred to as "data sets", but there has to be a simulator running on top of them to simulate some type of robot moving through the environment. Some details on this may help motivate why you dismiss VLFM. * Reproducibility: The results are all with (and contributions hinges on) GPT4, but the back-end of GPT4 changes substantially over time and there is afaik no way to run a specific version of GPT4 (only major versions). This means the numbers in the paper may not be reproducible at the time of publication. It would be helpful to also include results with a specific version of an open-source LLM to at least have something reproducible. -------- After rebuttal: The authors ran more experiments which addressed weaknesses #2 and #3. The quantitative improvements weren't as good as originally claimed but good enough. They even threw in a real robot experiment (video would also have been nice!)..The contribution is still a bit narrow but given the SOTA results on ObjNav it merits publication..

Questions

* Please clarify what kind of path planning and robot model you use, and in more detail why VLFM is not comparable. * Even assuming the VLFM "locomotion" is not comparable, could you not just use their 2D map component with your path planning? Both are 2D maps with scores.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Adequate except for the points above.

Summary

The paper proposes Geometric and Affordance Maps to tackle zero-shot object-goal navigation. It focuses on how to leverage VLMs to navigate toward objects of unfamiliar categories without prior training. The authors use GPT-4 to generate multiple attributes that describe affordance and geometry, and acquire attribute embeddings by CLIP text encoder. Besides, they propose a multi-scale scoring approach which divides each RGB image into multiple patches processed by CLIP visual encoder, to acquire multi-scale visual embeddings. Cosine similarity can be computed between those embeddings, resulting in a map used for navigation (e.g., applying Fast Marching Method based on the map). Experiments on HM3D and Gibson demonstrate improvements in success rates and success weighted by path length (SPL). The authors also provide an analysis on error pattern, which shows that detection error (missing the target object) is significantly reduced.

Strengths

The paper is clearly written. The proposed approach to leverage LLMs and VLMs to tackle zero-shot object-goal navigation is clean and simple.

Weaknesses

1. The overall approach is limited by LLMs and VLMs. The analysis in Sec. 5.9 (Table 4) implies that the improvement mostly results from the reduced detection error. Since off-the-shelf LLMs and VLMs are used, the technical contribution of the paper seems to be limited by this fact. Although geometric and affordance attributes are proved effective in Figure 3, the conclusion might be limited to the specific VLM used in this paper, namely CLIP. 2. It is better to replace Figure 3 with a table. It is not easy to tell the exact effect of $N_a$ and $N_g$. 3. Some sentences in Sec. 5.1 are with typos (perhaps in a rush). For example, "Gibson is generated by Al-Halah et al. [36] for agents with configuration A". And the citations in L214 are messy.

Questions

1. Can the authors compare the proposed method with PIVOT[1], which can also leverage VLMs to tackle robotic navigation? PIVOT focuses more on VQA, while this paper focuses on the similarity between the text and visual embeddings generated by CLIP. CLIP may not capture enough details, and thus the authors may need to introduce a multi-scale scoring approach. It is interesting to see whether VLM models can directly capture enough multi-scale information. 2. Can the authors also compare the time cost with baselines, like SemExp and VLFM? [1] PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have addressed the limitations in the appendix.

Response to Author Rebuttal

I thank the authors for doing the extra experiments, they make the paper stronger. I suggest you include the VLFM + motion planning results to make it more of an apples to apples comparison rather than just saying that VLFM is not comparable. I will raise my score pending reviewer discussion. That said, the improvements compared to VLFM + motion planning aren't very large (3-4% SR and 10-15% SPL) but they seem big enough to merit publication. The G+A approach is somewhat novel, although the other reviewers are right to question how specific to it is to CLIP. The entire affordance part also had rather marginal improvements as seen in the ablations (going from 1 -> 3 had no effect). Based on how limited CLIP language understanding tends to be, I had also not expected this to work well. If it is a good idea, maybe it will work better with some future VLM. As a tangent, it is not obvious what VLFM gained by training a custom PointNav policy instead of using an off-the-shelf motion planning algorithm like the authors. The VLFM paper does not seem to be clear on that. Any standard motion planner should be able to provide a near optimal (shortest) path for a simple robot like here (I assume it is point or sphere). It is curious that the original VLFM w/ trained-PointNav actually performed better than the version with the (optimal?) motion planner by the authors. Do the authors have any intuition of why it is more suitable for the ObjNav objective (or conversely why theirs is less so)?

Thanks!

Thanks for providing the clarifications. I am happy with your responses. Since most of my questions and weaknesses were clarifications, I will stick to my original rating.