We introduce MQ-Det, an efficient architecture and pre-training strategy design to utilize both textual description with open-set generalization and visual exemplars with rich description granularity as category queries, namely, Multi-modal Queried object Detection, for real-world detection with both open-vocabulary categories and various granularity. MQ-Det incorporates vision queries into existing well-established language-queried-only detectors. A plug-and-play gated class-scalable perceiver module upon the frozen detector is proposed to augment category text with class-wise visual information. To address the learning inertia problem brought by the frozen detector, a vision conditioned masked language prediction strategy is proposed. MQ-Det's simple yet effective architecture and training strategy design is compatible with most language-queried object detectors, thus yielding versatile applications. Experimental results demonstrate that multi-modal queries largely boost open-world detection. For instance, MQ-Det significantly improves the state-of-the-art open-set detector GLIP by +7.8% AP on the LVIS benchmark via multi-modal queries without any downstream finetuning, and averagely +6.3% AP on 13 few-shot downstream tasks, with merely additional 3% modulating time required by GLIP. Code is available at https://github.com/YifanXu74/MQ-Det.
Paper
Similar papers
Peer review
Summary
Based on recent vision-language fundamental models such as GLIP and Grounding DINO, the authors propose an improved multimodal query pipeline. A Gated Class-scalable Perceiver is used to apply cross-attention for both the language and vision query inputs. A masking strategy for text tokens is further proposed to improve the generalization ability. The proposed method achieves strong performance on zero-shot LVIS and ODinW benchmarks.
Strengths
- This paper first explores using both vision and language queries as inputs - The proposed method achieves superior experimental results over previous methods - The authors release the source code
Weaknesses
- Overclaimed training efficiency, see Q1 in the Questions section. The pre-training time of GLIP should also be considered, which makes MQ-GLIP more time and data-intensive. - Potential violation of zero-shot setting, see Q2 in the Questions section. The MQ-Det might improperly use exemplar images in a zero-shot setting, which is against the task definition. - Subpar full-shot performance, see Q3 in the Questions section.
Questions
1. I want to verify one point: the proposed method, such as MQ-GLIP, is trained from scratch, or finetuned from the initial weights from GLIP? According to #175-176: "we freeze the entire pre-trained detector and only train the newly-added gated class-scalable perceiver modules", I assume that MQ-GLIP is finetuned on the pre-trained GLIP model. Then I think the author overclaims their training efficiency. I think the pre-trained time of GLIP cannot be ignored. For example, in #232, it should be "MQ-GLIP-T adds extra 2% training time and extra 12% data usage of GLIP-T." rather than "only requires 2%". In my opinion, MQ-GLIP increases the training time and training data compared to GLIP, since MQ-GLIP brings an extra fine-tuning stage. 2. When evaluating the zero-shot setting such as zero-shot LVIS, does the MQ-Det requires the exemplar images for each class of the LVIS dataset? I guess the answer is 'yes' since the MQ-Det requires both the language and vision queries as inputs. But I think this way may violate the zero-shot setting. In the zero-shot setting, the LVIS classes are treated as unseen classes, and the exemplar images of the unseen classes are not allowed to be accessed. The model can only use the textual descriptions or text attributes associated with the unseen classes. So I think MQ-Det achieves strong zero-shot performance via a cheating inference evaluation. I think using the exemplar images of test-set classes is allowed for the few-shot setting but is not allowed for the zero-shot setting, which will weaken the contribution of this paper. 3. In table-2, the proposed MQ-Det performs worse than DINO-Swin-L on the full-shot setting. Can the authors provide some explainable reasons?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors discussed their limitations in section C of the appendix.
Summary
The paper propose MQ-Det, a novel module that integrates both language and visual queries efficiently for object detection tasks. This module enhances each category token with vision queries, providing rich detailed visual context to the text-based models. The proposed method is experimentally tested on LVIS and ODinW datasets, obtaining state of the art results. The method is versatile and can be applied to other state of the art language queried object detectors.
Strengths
The overall idea of the paper is easy to understand. The presented method, MQ-Det is novel because it introduces a unique combination of language and visual queries (multi-modal queries) to enhance object detection in an efficient manner. To train the proposed module it's not expensive and it does not require a lot of data. The proposed method, MQ-Det, was evaluated using two state of the art models, GLIP and GroundingDINO. The results demonstrated superior performance when MQ-Det was applied, showcasing the effectiveness of the MQ-Det approach.
Weaknesses
The comparison in the zero-shot and few-shot scenario is not fair since this method uses "5 instances as vision queries for each category from the downstream training set" thus having access to some information from the target dataset while all the other methods do not have any kind of access to that (for example owl-vit for image guided detection also doesn't use any fine-tuning, but when using vision queries from the target dataset they present the results as few-shot -- please see chapter 4.4 from owl-vit). Same observation for few-shot. Also it's not clear how are those 5 instances picked and how does that affect the final performance. The time and data comparison against state of the art it's not entirely fair since this method builds on top of GLIP and GroundingDINO. So this method benefits of the training data and training time of GLIP/GroundingDINO, thus the affirmations and comparisons may be misleading. It's not clear if the method can be used with text only or you always need some vision queries. Tied to that it's not clear what MQ-GLIP-T-Txt from Table 1 represents. Can you mask the visual queries? The comparison between MQ-GLIP-T-Txt and GLIP-T it's confusing. Various meta-parameters seem to be chosen randomly: for example choosing 5 as the vision queries. Ultimately, while I do see the merits of this work, I find the comparisons to be misleading and not entirely fair. minor typos: visual detials After rebuttal: The authors addressed my concerns properly and with the promised modifications I consider that this paper meets NeurIPS standards.
Questions
How are the vision queries chosen? How does this affect the performance? Choosing to have the related work chapter as the forth one it's unusual, is there any reason for that? please see Weaknesses.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
the paper discuss some of the limitations throughout the paper and does not address societal impact.
Summary
The paper introduces MQ-Det, a novel approach for open-vocabulary object detection that combines textual descriptions and visual exemplars as category queries. MQ-Det aims to address the limitations of existing text-queried object detectors by incorporating visual information and providing various granularity in the descriptions. The proposed architecture can be easily integrated with pre-trained language-queried detectors. To overcome the learning inertia problem caused by frozen detectors, MQ-Det employs a vision-conditioned masked language prediction strategy. This strategy involves randomly masking text tokens and allowing vision queries to independently predict objects, ensuring sufficient visual intervention during training. Overall, MQ-Det presents a simple yet effective architecture and training strategy that improves open-world object detection performance by incorporating multi-modal queries.
Strengths
1. The paper is well-written and easy to follow: The authors present their research in a clear and organized manner. 2. The paper tackles an important problem: The authors address the challenge of improving the robustness and generalization of current language-query based object detectors by incorporating visual cues. This is a significant problem in the field of open-vocabulary detection, as relying solely on textual descriptions can lead to insufficient granularity and ambiguous queries. By introducing multi-modal queries, the paper provides a promising solution to overcome these limitations and enhance detection performance in real-world scenarios. 3.The proposed GCP module and training techniques have broader applicability: GCP module introduced in the paper is a valuable contribution that can have implications beyond the scope of this study. For example, conditional gating layer of the GCP module provide a framework for effectively integrating useful visual information (instead of noisy visual templates) into language-queried detectors. Additionally, the random masking strategy addresses the learning inertia problem and enables better fusion of visual and textual cues. These findings can benefit future research. 4. The experiment is solid and proves the effectivenss of the proposed method.
Weaknesses
1. The training data in Table 1 and Table 2 is confusing. It appears that the authors loaded the pretrained GLIP models, which were pretrained on datasets such as Objects365, GoldG, CC4M, and Cap24M. However, the presented training data in those tables only includes the Objects365 dataset for proposed models. It is important to note that the performance of the model is achieved by utilizing all the mentioned pretraining datasets, rather than just the Objects365 dataset. Clarifying this discrepancy in the presentation of training data would provide a clearer understanding of the model's training process. 2. The evaluation of open-vocabulary or open-set object detection requires a clear separation of base and novel classes. Unfortunately, the paper does not explicitly evaluate the performance on these two sets separately. This omission makes it difficult to assess the model's generalization capability specifically on novel classes. To gain a deeper and better understanding of the model's performance in generating novel classes, it is recommended to conduct separate evaluations and present the results accordingly.
Questions
As mentioned in the paper, the selection of visual templates during inference plays a critical role in the success of the proposed approach. I would like to inquire about the impact of random sampling templates on the performance. Specifically, I am interested in whether the authors performed multiple inference runs, averaged the performance results, and presented the inference variance with error bars. Could you please clarify whether this analysis was conducted and whether the paper provides information on the effect of random sampling templates on performance?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
I do not see potential negative societal impact.
Summary
In this paper, the authors proposed MQ-Det which leverage both textual and visual information for object detection in the wild. The proposed plug-and-play GCP module is very compatible with existing mainstream architectures. The authors conducted extensive experiments on multiple benchmark datasets and showed improved performance over several baseline methods for both zero-shot and few-shot detection.
Strengths
1. Leveraging both textual and visual info for open-vocabulary detection makes lots of sense as they provide different level of signals for recognizing objects as mentioned in the draft. The proposed GCP module and masking based training strategy is a novel technique to combine those informations. 2. The authors conducted extensive experiments and ablation studies to sufficiently validate the proposed approach over multiple baseline methods on several benchmarking datasets. 3. Writing is good and easy to follow. The tables and figures are also easy to understand.
Weaknesses
1. For the last row of Tab. 3 (b) where the joint input leads to lower performance, the authors' explanation is that "this task may introduce redundant information and rise the learning difficulty." This is not very clear to me since redundancy could also lead to easier training rather than difficulty. Could you please elaborate more on this? 2. For Fig. 4, it is weird that the baseline 'No Vision Query' (blue box) model actually recognize the two zebras as either elk or horned cow. These two zebras looks very easy to recognize even without visual cue for large VL models. Is this due to insufficient training of baseline models or hand-picked examples to illustrate the idea? 3. As illustrated in Fig. 3, with increasing amount of training data, the gap between MQ-Det and other baseline methods are getting more and more closer. What would be a rule of thumb strategy for using MQ-Det versus other simpler methods?
Questions
Please refer to the weaknesses section for details.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
NA
I appreciate the authors' efforts in addressing the concerns through their rebuttal and the supplementary experiments. I recognize the merits of the proposed method and commend the clarity of the manuscript's presentation. However, my reservations persist regarding the fairness of the zero-shot comparisons. Despite the authors' acknowledgment of this issue, the forthcoming modifications may only partially alleviate the reader's uncertainties. It's pertinent to note that certain methods, even in a zero-shot paradigm without any fine-tuning or weight updates, might inherently benefit from the availability of 5 samples, as exemplified by the "owl-vit" method.
Thanks, authors for the detailed response. It largely solves my concern. Thus, I keep my original rating in the current stage.
Thanks for the reviewer's recognition and constructive suggestions.
Thanks to the authors for providing the rebuttal replies. > Training efficiency on time and data The authors agree that it should be “add extra” rather than “only requires” in #232. In Table 1, the authors may consider adding two columns, e.g., ‘training time’ and 'modulating time’, ‘training data’ and ‘modulating data’ to avoid confusion. > Zero-shot setting While the authors acknowledge that their experiments do not conform to the zero-shot setting, they introduce a new term, the "finetuning-free setting". However, my concerns remain. For instance, the comparison in Table 1 is not fair because other methods did not use the 5 vision examples. Also, since the authors propose a new setting, they may need more effort to provide more baselines rather than the variants of MQ-GLIP-T in Appendix A.1 under their new setting. Reviewer 9Uan also expressed concerns about this particular point. I also have the same concern with the reviewers 9Uan about missing the ablation studies about the number of vision examples used in the "finetuning-free setting". > Full-shot performance I am still confused about why the authors do not provide the MQ-GLIP-L results in the few-shot/full-shot setting. Overall, I will keep my initial rating.
Thanks for the reviewer's response. Our new response is as follows. > Q1: In Table 1, the authors may consider adding two columns, e.g., ‘training time’ and 'modulating time’, ‘training data’ and ‘modulating data’ to avoid confusion. Thanks for the reviewer's valuable recommendation. We will revise the table according to your suggestion. > Q2: For instance, the comparison in Table 1 is not fair because other methods did not use the 5 vision examples. Also, since the authors propose a new setting, they may need more effort to provide more baselines rather than the variants of MQ-GLIP-T in Appendix A.1 under their new setting. 1. The finetuning-free comparison between our multi-modal queried method and previous language-queried methods is reasonable because it verifies the superiority of multi-modal queries over singe-modal queries, and shows that multi-modal queried object detection can be a promising future direction, rather than simply presenting a state-of-the-art model. 2. Thanks for the reviewer's suggestion. It is hard to find existing baselines that suitable in the multi-modal queried object detection setting, as recognized in the reviewer's initial review: "This paper first explores using both vision and language queries as inputs". To this end, we modify GLIP/GroundingDINO through the method in OWL-ViT as baselines and conduct finetuning-free evaluation in LVIS MiniVal. Specifically, our modification contains three steps. 1) Acquiring naive vision quries: we feed the visual examplars togather with LVIS 1203 category texts into GLIP/GroundingDINO, then crop the corresponding regions on the output image features using an RoIPooler. The averaged cropped region features are treated as vision queries. 2) Constructing naive multi-modal queries: we average the classification logits of original language queries and the naive vision queries as the results of multi-modal queries. 3) Detection evaluation: we separately use the original language queries, the naive vision queries, and the naive multi-modal queries for evaluation on LVIS MiniVal. The classification is conducted via dot-product similarity that similar to OWL-ViT. The results are shown in the following table. For example , GLIP-T, GLIP-T-Img, and GLIP-T-MM denote GLIP with original language-queries, naive vision queries, and naive multi-modal queries, respectively. The number of vision queries is set to 5. | Model | $AP$ | $AP_r$ | $AP_c$ | $AP_f$ | | ------------------- | ---- | ------ | ------ | ------ | | GroundingDINO-T | 25.7 | 15.2 | 21.9 | 30.9 | | GroundingDINO-T-Img | 7.7 | 2.6 | 6.5 | 9.8 | | GroundingDINO-T-MM | 14.7 | 8.2 | 12.6 | 17.9 | | MQ-GroundingDINO-T | 30.2 | 21.7 | 26.2 | 35.2 | | GLIP-T | 26.0 | 20.8 | 21.4 | 31.0 | | GLIP-T-Img | 7.6 | 2.4 | 6.8 | 9.5 | | GLIP-T-MM | 15.4 | 10.6 | 13.4 | 18.0 | | MQ-GLIP-T | 30.4 | 21.0 | 27.5 | 34.6 | | GLIP-L | 37.3 | 28.2 | 34.3 | 41.5 | | GLIP-L-Img | 10.9 | 4.1 | 9.2 | 13.7 | | GLIP-L-MM | 24.3 | 17.7 | 21.5 | 27.9 | | MQ-GLIP-L | 43.4 | 34.5 | 41.2 | 46.9 | The results show that directly using vision queries in language-queried detectors through some naive modification demonstrates poor detection performance, and combining such vision queries with language queries as multi-modal queries impairs the performance. > Q3: I also have the same concern with the reviewers 9Uan about missing the ablation studies about the number of vision examples used in the "finetuning-free setting". We have provided the ablation results in the rebuttal. Please refer to the `Meta-parameters and the related work chapter` part of our initial response to Reviewer 9Uan. > Q4: I am still confused about why the authors do not provide the MQ-GLIP-L results in the few-shot/full-shot setting. We have provided the results in the paper. Please refer to Figure 3 in the paper and Table Ⅲ in the Appendix. We did not present the MQ-GLIP-L results in Table 2 only for fair comparison. --- Please feel free to let us know if you have other questions.
How are the results changing in the zero-shot setup if only one example is used as opposed to 5?
Thanks for the reviewer's response. Our new response is as follows. > Q1: It's pertinent to note that certain methods, even in a zero-shot paradigm without any fine-tuning or weight updates, might inherently benefit from the availability of 5 samples, as exemplified by the "owl-vit" method. 1. In fact, "owl-vit" did not demonstrate the benefits from vision samples compared to its language-queried counterpart. In chapter 4.4 of the "owl-vit", only the results of using 1 vision sample and 10 vision samples were compared, without including its language-queried results. To verify this, we conducted experiments using its code in Huggingface and compared its language-queried model, OWL-ViT-Txt, with its vision-queried model, OWL-ViT-Img, as shown in the table below. OWL-ViT-Img, MQ-GLIP-T, and MQ-GLIP-T-Img all use one identical vision sample as the vision query. The results show that the usage of vision samples alone in OWL-ViT showcases inferior open-world detection performance. Furthermore, "owl-vit" does not support language and vision queries as joint inputs, which neglects the complementary nature of vision and language, as outlined in our related work chapter. | Model | Rabbit | Pothole | ODinW-13 | | -| ------ | ------- | -------- | | OWL-ViT-Txt (ViT-L/14) | 73.0 | 17.5 | 40.9 | | OWL-ViT-Img (ViT-L/14) | 19.4 | 0.3 | 10.5 | | MQ-GLIP-T | 74.4 | 13.2 | 43.9 | | MQ-GLIP-T-Txt | 71.6 | 6.7 | 41.9 | | MQ-GLIP-T-Img | 71.1 | 4.0 | 29.6 | 2. We are the first work to explore multi-modal queries in object detection, which supports jointly using textual descriptions and visual exemplars. The finetuning-free comparison between our multi-modal queried method and previous language-queried methods is reasonable because it verifies the superiority of multi-modal queries over singe-modal queries, and shows that multi-modal queried object detection can be a promising future direction, rather than simply presenting a state-of-the-art model. 3. It is challenging to use multi-modal queries without any finetuning for previous language-queried detectors. We conduct further experiments to show that, directly using vision queries in language-queried detectors through some naive modification demonstrates poor detection performance, and combining such vision queries with language queries as multi-modal queries impairs the performance. We modify GLIP/GroundingDINO through the method in OWL-ViT as baselines and conduct finetuning-free evaluation on LVIS MiniVal. Specifically, our modification contains three steps. 1) Acquiring naive vision quries: we feed the visual examplars togather with LVIS 1203 category texts into GLIP/GroundingDINO, then crop the corresponding regions on the output image features using an RoIPooler. The averaged cropped region features are treated as vision queries. 2) Constructing naive multi-modal queries: we average the classification logits of original language queries and the naive vision queries as the results of multi-modal queries. 3) Detection evaluation: we separately use the original language queries, the naive vision queries, and the naive multi-modal queries for evaluation on LVIS MiniVal. The classification is conducted via dot-product similarity that is similar to OWL-ViT. The results are shown in the following table. For example , GLIP-T, GLIP-T-Img, and GLIP-T-MM denote GLIP with original language-queries, naive vision queries, and naive multi-modal queries, respectively. The number of vision queries is set to 5. | Model | $AP$ | $AP_r$ | $AP_c$ | $AP_f$ | | -| ---- | ------ | ------ | ------ | | GroundingDINO-T | 25.7 | 15.2 | 21.9 | 30.9 | | GroundingDINO-T-Img | 7.7 | 2.6 | 6.5 | 9.8 | | GroundingDINO-T-MM | 14.7 | 8.2 | 12.6 | 17.9 | | MQ-GroundingDINO-T | 30.2 | 21.7 | 26.2 | 35.2 | | GLIP-T | 26.0 | 20.8 | 21.4 | 31.0 | | GLIP-T-Img | 7.6 | 2.4 | 6.8 | 9.5 | | GLIP-T-MM | 15.4 | 10.6 | 13.4 | 18.0 | | MQ-GLIP-T | 30.4 | 21.0 | 27.5 | 34.6 | | GLIP-L | 37.3 | 28.2 | 34.3 | 41.5 | | GLIP-L-Img | 10.9 | 4.1 | 9.2 | 13.7 | | GLIP-L-MM | 24.3 | 17.7 | 21.5 | 27.9 | | MQ-GLIP-L | 43.4 | 34.5 | 41.2 | 46.9 | > Q2: How are the results changing in the zero-shot setup if only one example is used as opposed to 5? Please refer to the `Meta-parameters and the related work chapter` part of our initial response for the results of MQ-GLIP-T on LVIS MiniVal. Also, the MQ-GLIP-T and MQ-GLIP-T-Img in the first table uses one example for ODinW-13 evaluation in the zero-shot setup, namely, | Model | ODinW-13 (1 vision query) | ODinW-13 (5 vision queries) | | -| - | -| | MQ-GLIP-T | 43.9 | 45.6 | | MQ-GLIP-T-Img | 29.6 | 31.9 | --- If you have other concerns, please feel free to reply.
I appreciate the clarity and structure of the paper and find the proposed method to be innovative. My primary concern is ensuring the clarity of the zero-shot setup for readers. I've noted your intention to replace "zero-shot" with "finetuning-free," which I believe is a step in the right direction. Can you confirm that this terminology will be consistent throughout the manuscript? Furthermore, I suggest adding a column in Tab.1 that clearly indicates the number of examples each method utilizes without finetuning. This won't detract from the strengths of your paper, as it stands robustly on its own merits. I simply believe this will aid readers in grasping the setup more effortlessly. I'm inclined to enhance my review score if we can achieve consensus on this "finetuning-free" presentation aspect.
We thank the reviewer's recognition and valuable suggestions. We will thoroughly revise the paper for the clarity of the "finetuning-free" presentation. > **We will ensure the consistency in the usage of the terminology "finetuning-free" throughout the manuscript. The detailed modifications that we will make are listed as follows.** Generally, we will add a clear defination on "finetuning-free" in the introduction and experiment chapters. We will also thoroughly revise the details in the paper as follows. ***Abstract*:** **Lines 13-16:** revise the sentence to "For instance, MQ-Det significantly improves the state-of-the-art open-set detector GLIP by +7.8% AP on the LVIS benchmark via multi-modal queries without any downstream fine-tuning, ... , with merely additional 3% modulating time upon GLIP." ***Introduction*:** **Lines 50-53:** add the definition of the "finetuning-free" setting and revise the sentence to "We evaluated our models on a finetuning-free setting, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. With only one epoch ... , our approach impressively improves the finetuning-free performance by +7.8% on the LVIS benchmark through providing the model with 5 visual exemplars along with textual category descriptions." **Lines 74-78:** revise the sentence to "Our proposed MQ-Det demonstrates powerful transferability on the finetuning-free and few-shot scenarios, ... Specifically, MQ-Det outperforms GLIP by +7.8% AP in the finetuning-free detection on the challenging LVIS [13] and averagely +6.3% AP on 13 downstream few-shot detection tasks [23], modulated with merely 3% of the training time required by GLIP." ***Experiments*:** ***Chapter 3.1.2*:** **Lines 218-220:** highlight the definition of the "finetuning-free" setting and revise the sentence to "We also evaluate our method in a finetuning-free setting, namely, users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. During finetuning-free evaluation, we extract 5 instances as vision queries for each category from the downstream training set without any finetuning." ***Chapter 3.2.1*:** **Modify the chapter name**: "Multi-modal queried detection without finetuning". **The begining of line 226:** add the definition of the "finetuning-free" setting: "We evaluate the model's ability ... in a finetuning-free setting, where users can detect their customized objects through textual descriptions, visual exemplars, or both without any fine-tuning. " **Lines 228-231:** "Overall, MQ-Det demonstrates strong finetuning-free transferability with ... MQ-GLIP-L surpasses the current SoTA on LVIS by a large margin with simply 5 visual exemplars provided, reaching ..., which verifies the superiority of our multi-modal queries over single-modal queries. Meanwhile, MQ-Det demonstrates good training efficiency, e.g., MQ-GLIP-T only requires additional 2% training time and 12% data usage of GLIP-T when modulated upon GLIP-T. It is worth noting that modulating upon pretrained GLIP/GroundingDINO indirectly utilizes their pretraining data. The efficiency here describes that our approach allows current mainstream language-queried detectors to be equiped with multi-modal queries only through a lightweight modulating process, avoiding training from scratch." ***Conclusion*:** **Lines 343**: "MQ-Det shows promising gains on the finetuning-free and few-shot settings ..." ***Others*:** We will revise Table 1 and Table 2 based on the reviewer's suggestions. We will replace the "zero-shot" in the title of Table 3 with "finetuning-free". Similar revisions will be made in the appendix. > **We will modify Table 1 following the reviewer's suggestion. For example:** Table 1: Finetuning-free detection results on the LVIS benchmark. Differently, we provide models with 5-shot instances as vision queries along with the language queries without any fine-tuning. ... $^\dagger$ Modulating upon pretrained models indirectly utilizes their pre-training data, and consumes more time if we take the training time of the pretrained language-queried detectors into consideration. | Model| ... | # Vision Queries | ... | Pre-train Data| ... | Training Time | ... | | -| -| -|-|-|-|-|-| | ... | ... | ... | ... | ... | ... | ... | ... | | GLIP-T | ... | 0 | ... | O365, GoldG, Cap4M | | 480 | ... | | ... | ... | ... | ... | ... | ... | ... | ... | | MQ-GLIP-T | ... | 5 | ... | O365 (+GLIP$^\dagger$) | ... | 10$^\dagger$ | ... | | MQ-GroundingDINO-T | ... | 5 | .. | O365 (+GroundingDINO$^\dagger$) | ... | 10$^\dagger$ | ... | --- Please feel free to reply if you have further suggestions or questions.
I appreciate the proposed changes and believe they will enhance the clarity of the paper. I've revised my score to 'Accept'.
Thanks for the reviewer's recognition and raising your score. Your suggestions significantly help improve the quality of our paper.
Thanks for the responses. My main concerns are addressed and thus keep my original positive rating.
Decision
Accept (poster)