3DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o

We appreciate the reviewer’s thoughtful feedback ### w1: How are the 3D axes set? Does the origin always lie in the same corner of the scene? What if the axes are not well aligned with the scene's boundaries? In Figure 3-3, I noticed that the output of Y(=7.08) seems not very accurate compared with the other aligned ones. The 3D axes are aligned with a principal component of the input point cloud. On the scannet, if the axes are not well aligned with the scene's boundary, the performance will significantly drop. ### w2: The idea is very interesting. However, I still need to point out that the runtime of the model inference is not efficient enough for the practical problems (planning, localization, etc.) discussed in the paper. The runtime is indeed a limitation for practical problem-solving. We will add a discussion of the limitations in the final paper. ### w3: Ball positioning experiments We conduct the point positioning task as shown in [Figure 1](https://github.com/Anonymous123-ctrl/3DAxisPrompt540/blob/main/point_position.png). The ball position can be acquired by direct simple question "Can tell me the position of the ball from the images?" and the input prompted images. We test about 20 times and the results are as follows. | 3D Axis |Mark Type | Prompt Elements | Metric| |---------|-----------|-------------------|---------| Scannet | - | - | To center ~ | 3D Mark | Mark | N/A | Yes| baseline | - | 0.391 | Yes| 2D Mark | Mark+2D contour (colors) + CoT | 0.219 | Point Positioning | - | - | To center Yes| 2D Mark | Mark+2D contour (colors) + CoT | 0.184 From the results, we can see that the ball localization is more accurate than the object localization. Furthermore, we from all the other experiments and the investigation. We find that the MLLM can utilize these localization abilities. In other words, the localization abilities help promote the 3D grounding abilities, as shown in the comparison experiments with LLM-Grounder. ### w4: other scaled axis We are working on these experiments and results will be provided soon.

We thank the reviewer for the insightful feedback. ### w1: comparison on widely accepted benchmarks We add more comparision experiments with LLM-Grounder on the ScanRefer dataset as follows: |Method | Visual Grounder | LLM Agent | Acc@0.25 | Acc@0.5| |--------|-----------------|------------|------------|---------| ScanRefer | - | - | 35.2 | 20.7 LERF | LERF | - | 4.5 | 0.3 LLM-Grounder | LERF | GPT-4 | 7.0 (+2.5) | 1.6 (+1.3) Ours | LERF + 3DAxisPrompt | GPT-4o | 9.1 (+4.6) | 3.1 (+2.8) Specifically, we replace the LLM Grounder with the proposed 3DAxisPrompt+GPT-4o. Each QA, the Input Text prompt of LLM-Grounder and the observation images are set as input to GPT-4o, the rest remains the same. From the results, we can see that the 3D AxisPrompt significantly improves the performance of the grounding performance of LERF and LLM-Grounder. This shows that the 3DAxisPrompt plus the text prompt of LLM-Grounder can significantly promote grounding abilities. The reason that we choose 3D localization is that the localization ability is only provoked by the proposed 3DAxisPrompt and we think it can represent the 3D grounding abilities in some way. The LLG-Grounder is very inspiring to us. Therefore, we add more comparision results follows the LLM-Grounder ### w2: in-depth analysis It is important to align the x or y axis with the main direction of the scene. On scannet, the main direction is the x or y of the floor. In fact, we develop a simple algorithm in our observation image rendering tool to determine the direction by calculation the Principal component of the input point cloud, code will be publised if accept. There is a systematic study on the view selection. However, to eliminate the view selection, we choose to render multiview images and set the series of images as the input visual prompt, as shown in [Folder](https://github.com/Anonymous123-ctrl/3DAxisPrompt540/tree/main/pipeline/ObservationImagesby3DAxisPrompt). Zooming in will improve localization accuracy. Also, some text prompt regarding the coarse position description in the instruction will also improve the performance. We will add a comprehensive conclusion of details like these in the final paper.

Thanks for the constructive suggestions. ### w1:Limited contribution: However, the paper lacks scientific investigation into why MLLM can perform 3D grounding with the proposed prompting and does not provide a novel perspective on the problem. The paper’s structure resembles a technical report rather than a conference contribution. We have to restate that we think the idea is worth cherishing. The 3D grounding ability is indeed promoted by a simple and easy-to-deploy prompt. Also, we add more comparision experiments with LLM-Grounder on the ScanRefer dataset as follows: |Method| Visual Grounder | LLM Agent | Acc@0.25 | Acc@0.5| |---------|--------------------|--------------|-------------|-----------| ScanRefer | ~ | ~ | 35.2 | 20.7 LERF | LERF | ~ | 4.5 | 0.3 LLM-Grounder | LERF | GPT-4 | 7.0 (+2.5) | 1.6 (+1.3) Ours | LERF + 3DAxisPrompt | GPT-4o | 9.1 (+4.6) | 3.1 (+2.8) Specifically, we replace the LLM Grounder with the proposed 3DAxisPrompt+GPT-4o. Each QA, the Input Text prompt of LLM-Grounder and the observation images are set as input to GPT-4o, the rest remains the same. From the results, we can see that the 3D AxisPrompt significantly improves the performance of the grounding performance of LERF and LLM-Grounder. This shows that the 3DAxisPrompt plus the text prompt of LLM-Grounder can significantly promote grounding abilities. Also, we consider the visual prompting process as introducing lost 3D information when capturing the 2D observation images. Following this thought, we conduct comprehensive investigation on the potential prompt format as in Section 3.2 and 3.3 and the supplement section. As for why the MLLM can perform 3D grounding with the prompting. First, we conduct the ablation study and find out the axis, ticks, and labels are all important in promoting 3D grounding. Based on the ablation study and the investigation on the potential prompt format, we try to conclude our perspective on the 'why' question, as follows: 1. Initial grounding abilities are found in the LMM, such as knowing the cup is on the desk, from the 2D images. However, this grounding ability is limited to the relative position relationship. 2. By adding the proposed visual prompts, we think it acts like a ruler or anchor for the scene. Thus, based on the relative position relationship sensing abilities, MLLM can find the object relative position to the axis, thus producing the 3D localization results. 3. To support our findings, it can be seen from the ablation study that the 3DAxisPrompt totally fails when removing the ticks of the axis. This shows that the numbers that indicate the absolute position relationship is essential and MLLM relies on the anchor to sense the 3D position from the 2D observation images. Also, when other elements of the axis are removed, 3DAxisPrompt can still work but the performance drops a little bit (3.5%). This also implies the importance of ticks. 4. In conclusion, the MLLM provoke the 3D grounding (esspecially 3D localization) based on the absolute position achored by the axis and the relative position relationship sensing. ### w2: Evaluation issues: We enrish the Table 1 in the paper as shown below. We add the first part (adding 3D axis). Also, to better interpret the numbers in the table, we add the results of only using 3D axis without additional marker as baseline. As for no axis, the MLLM fails to provoke 3D localization. |3D Axis |Mark Type | Prompt Elements | Scannet | Scannet| |---------|-------------|----------------------|-----------|----------| ~ | - | - | To center | To bbx No | 3D Mark | Mark | N/A | N/A Yes| baseline | - | 0.391 | 0.296 Yes| 3D Mark | Mark | 0.333 | 0.216 Yes| 3D Mark | Mark+OBB | 0.350 |0.231 Yes| 3D Mark | Mark+AABB (red) |0.376 |0.219 Yes| 3D Mark | Mark+AABB (colors) |0.311 |0.207 Yes| 3D Mark | Mark+3D edge points |0.305 |0.205 Yes| 2D Mark |2D contour (colors) | 0.320|0.175 Yes| 2D Mark | Mark+2D contour (colors) | 0.271 | 0.138 Yes| 2D Mark | Mark+2D contour (colors) + CoT | 0.219 | 0.115 ### w3: Scientific validity and significance we add more comparison experiments with LLM-Grounder on the ScanRefer dataset as follows: |Method | Visual Grounder | LLM Agent | Acc@0.25 | Acc@0.5| |--------|-----------------|------------|------------|---------| ScanRefer | - | - | 35.2 | 20.7 LERF | LERF | - | 4.5 | 0.3 LLM-Grounder | LERF | GPT-4 | 7.0 (+2.5) | 1.6 (+1.3) Ours | LERF + 3DAxisPrompt | GPT-4o | 9.1 (+4.6) | 3.1 (+2.8) ### w3: Typo: Truly sorry for the mistakes, typos will be rechecked.

We are truly sorry for the delayed reply. ### w1: To what extent does this method improve some 2D LLMs, such as LLAVA-NEXT? In addition to the benchmarks listed in the paper, can this method be applied to more benchmarks? I need more confidence and evidence of improvements on widely recognized benchmarks. We are trying our best to conduct comparision experiemtns with LLM-Grounder on ScanRefer dataset, the comparision experiments also mentioned by other reviewers. The workload is massive. Now we provide the following additional results on ScanRefer: Method | Visual Grounder | LLM Agent | Acc@0.25 | Acc@0.5 ----------|-------------------|--------------|-------------|----------- ScanRefer | - | - | 35.2 | 20.7 LERF | LERF | - | 4.5 | 0.3 LLM-Grounder | LERF | GPT-4 | 7.0 (+2.5) | 1.6 (+1.3) Ours | LERF + 3DAxisPrompt | GPT-4o | 9.1 (+4.6) | 3.1 (+2.8) Specifically, we replace the LLM Grounder with the proposed 3DAxisPrompt+GPT-4o. Each QA, the Input Text prompt of LLM-Grounder, and the observation images are set as input to GPT-4o; the rest remains the same, as shown in [Figure 1](https://github.com/Anonymous123-ctrl/3DAxisPrompt540/blob/main/vs_llm-Grounder.pdf). We add an anonymous link for display. From the results, we can see that the 3D AxisPrompt significantly improves the grounding performance of LERF and LLM-Grounder. This shows that the 3DAxisPrompt plus the text prompt of LLM-Grounder can significantly promote grounding abilities. We are still working on more comparison experiments. ### w2: This method seems a bit too simplistic; please restate its innovativeness and necessity, as well as how it differs from similar approaches. If necessary, more visualizations can be provided. Our initial attempt is to expand the 2D visual grounding abilities to 3D visual grounding abilities in the vision-language model (VLM) by the proposed visual prompts. We choose the 3D localization ability of the 3D grounding because it is challenging for VLM (such as GPT-4o) to invoke 3D grounding ability (especially 3D localization) without finetuning or network changes. We think the 3D localization from 2D images is a strong point to support our statement "promote the 3D grounding ability". Furthermore, the visual prompts can be seen as introducing more 3D information lost when capturing the observation images from the 3D scene. Following this thought, we designed the visual prompt scheme (3DAxisPrompt). The innovation is that the vision language model (such as GPT-4o) struggles to process the 3D grounding abilities from 2D images without finetuning or modifying the network. Notably, we are not able to play with the mega model like the GPT series. However, by preprocessing the input images with the visual prompt scheme, the model can show more powerful 3D grounding abilities (especially the 3D localization through 2D images). We think this significantly makes these mega models like GPT-4o more powerful. The necessity is that we consider the Large Multi-modal Model (LMM) as an interaction tool that can interrelate humans and their surrounding environments. GPT-4o, as one of the most powerful AI (LMM), should be able to possess the potential to become the interaction tool. However, 3D localization from 2D images or other problem-solving abilities necessitate this localization ability, which is challenging for GPT-4o right now. We think it is necessary to empower the LMM with more powerful 3D grounding abilities so that it can help solve more tasks based on the promoted 3D grounding abilities. We provide more visualized pipeline as shown in [Figure 2](https://github.com/Anonymous123-ctrl/3DAxisPrompt540/blob/main/pipeline.pdf) and more intermedia results in the [Folder](https://github.com/Anonymous123-ctrl/3DAxisPrompt540/tree/main/pipeline) [1] LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent. ICRA 2024