We thank you for your valuable and thoughtful feedback.
We are encouraged that you find our paper novel, effective, and reasonable and that our experiments are comprehensive with state-of-the-art results.
Below, we will address your concern and incorporate all the feedback.
> 1: In Alg.1, FIND function maps a localized object, a relation and a set of candidate objects to an output localized object. What is the detailed process of FIND? Is the relations between objects known?
The relations between objects are not known. Thus, we have to extract them.
First, we extract the entire mentioned objects and their relations from the utterance using the scene graph parser [[1]](https://aclanthology.org/W15-2812.pdf), [[2]](https://openaccess.thecvf.com/content/CVPR2023/papers/Wu_EDA_Explicit_Text-Decoupling_and_Dense_Alignment_for_3D_Visual_Grounding_CVPR_2023_paper.pdf).
Our method involves iterating over all anchors extracted from the utterance and searching for candidate objects in P that share the same class.
When the anchor class is represented singularly in the scene, we return the matched object.
However, we run the FIND function in scenarios where disambiguation is required due to multiple objects belonging to the anchor class.
The FIND function leverages the parsed spatial relations and the localized objects, so far, within the scene to identify the remaining anchors accurately.
However, it is not guaranteed that the FIND function will be able to localize the remaining unlocalized anchors accurately.
Thus, in this case, as shown in the last step in Algorithm 1, we randomly sample an object of the same class.
> 2: Alg.1 receive the input of object proposals P. Is it the output from the 3D object detection model? If so, the anchor localization may be bottlenecked by the object proposals.
The prominent role of the localization module is to assign an object proposal for each anchor mentioned in the input description without requiring human effort.
A pertinent example can be drawn from the labeling procedure employed in PhraseRefer, which demanded a cumulative workforce commitment of 3664 hours.
Thus, we use the GT object proposals P from the ScanNet dataset, following the related works, e.g., [ReferIt3D](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460409.pdf).
Hence, given a new dataset, our algorithm requires two pieces of information: bounding boxes (GT or predicted) and at least one localized object.
Then, our algorithm works iteratively, starting from prior knowledge, i.e., at least one localized anchor/target, to localize the rest of the mentioned anchors in the utterance given the previously localized ones without the need for any human efforts.
To show the influence of this module, we report the results for two variants for each baseline, one using our pseudo-noisy labels (Algorithm 1), which is human-free, and the second one using the anchors GT annotations from “PhraseRefer.”
Significance performance gain can be shown for the MVT case in Table 4, where using our pseudo labels, we outperform “PhraseRefer” and “ScanEnts,” which utilize GT anchors annotations.
Furthermore, using their manual annotations instead of our pseudo labels, we achieve SOTA results and outperform them significantly.
This emphasizes the importance of our proposed approach.
> 3: Considering the elaborate process of pseudo label generation, the propose frameworks seems not "data efficient" in data preparation, though it is data efficient in 3D grounding training.
We believe our answers to the previous two questions should answer your concerns and alleviate any misunderstanding about how our pseudo-label generator works.
If any more clarifications are needed, please let us know; we would love to elaborate more.
> 4: The interpretability is limited. From the input end, GPT-3.5 is used for extract the logical order of objects given an input utterance, thus adding interpretability. But the logical relations of the objects is not explicit modeled. Instead, it is implicit learned in the Seq-2-Seq network.
We argue that modeling the relations implicitly is enough, as to predict the logical order of objects correctly is a much more complex problem.
To solve it, the model has to extract the relations correctly first.
To probe our claim, we conducted an experiment where we added a new training objective for the language head.
Instead of only predicting the target of the anchors from the input utterance, we add another loss on predicting the relationship between the objects as pairs and use [bipartite loss](https://arxiv.org/abs/2005.12872).
However, as shown in the following Table, there is no gain from learning the relations explicitly.
| **Method** | **Grounding Accuracy** |
|----------|------------------------|
| **MVT** | 55.1 |
| **MVT+CoT** | **64.4** |
| **MVT+CoT+Realtions Objective** | **64.2** |