Detecting objects based on language information is a popular task that includes Open-Vocabulary object Detection (OVD) and Referring Expression Comprehension (REC). In this paper, we advance them to a more practical setting called Described Object Detection (DOD) by expanding category names to flexible language expressions for OVD and overcoming the limitation of REC only grounding the pre-existing object. We establish the research foundation for DOD by constructing a Description Detection Dataset ($D^3$). This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission. By evaluating previous SOTA methods on $D^3$, we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi-functional methods also do not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code are available at https://github.com/shikras/d-cube and related works are tracked in https://github.com/Charles-Xie/awesome-described-object-detection.
Paper
References (62)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper presents a new multi-modal computer vision task, called Described Object Detection, which is a superset of existing OVD and REC tasks. In particular, the DOD task seeks to create models which can detect multiple instances of something in images, from textual descriptions, which could include describing the absence or presence of something. The paper then goes on to create a new dataset for this task, based on an existing one, called $D^3$. It then shows how existing OVD, REC, Bi-Functional, and proposed a baseline method perform on the data set.
Strengths
The paper is attacking a significant problem, is of high experimental quality, and is novel. The idea of having free-text descriptors for finding things in images is extremely important to the adoption of ML for various important, real-world tasks. In fact, I was surprised to learn that given the importance of this concept, something like DOD had not been proposed before. The paper is also of high quality in that it not only identifies this shortcoming of real-world, open-set detection tasks (i.e. DOD), but also creates a dataset, tests it across a range of existing methods, and proposes a new baseline. Because the paper does propose a new formal problem definition for something that is desired in the real-world performance of ML systems, and therefore realizes how current tasks like REC and OVD fall short of what is actually needed for things like open-set, zero-shot object detection it is a novel look at the problem domain. Finally, it is also worth mentioning that the inclusion of the absence examples is a novel idea, which shows in the poor performance of models on absence instances in the data set.
Weaknesses
The only major weakness of the paper is its clarity. This lack of clarity manifests primarily in two places. First, the description of the creation of the data set needs more detail. Specifically, how was CLIP utilized for adding complete annotations? Were full images given to CLIP with all of the possible annotations and then the most probable selected as an annotation for the image? And, if so, how did you deal with multiple annotations being present in the same images? Additionally, how were the negative image annotations created? Were these done by hand, and, if so, by how many people? Similarly, how many people were involved with creating multiple-instance bounding boxes? The annotation section might benefit from a diagram to better explain this process. Second, the baseline method is poorly described. It's not clear from the main paper or the semantic similarity how the different proposed components come together into the OFA-DOD model. The paper would greatly benefit from a figure displaying the OFA model and the OFA-DOD model to better understand the architecture of the OFA-DOD model. As it stands now, a reader would have a hard time recreating the OFA-DOD model from the given text. Finally, there are some minor typo issues that need to be addressed. For example, line 130 page 4 repeats “their” twice, and line 212 on page 8 has the wrong subject-verb agreement (i.e. “fails” instead of “fail”). The paper could use another proofread for typos and grammar.
Questions
Please see the weaknesses section on the questions related to the annotation process and the OFA-DOD model.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
2 fair
Contribution
3 good
Limitations
The paper adequately addresses its limitations and potential negative societal impact.
Summary
In this paper, the authors propose the definition, dataset, evaluation metrics and benchmark results of a new task - Described Object Detection (DOD). DOD is designed to detect objects aligned with full, presence or absence langauge descriptions. The proposed evaluation metrics have three groups based on full, presence or absence descriptions. The authors also propose the Intra-scenario mAP that only detect the categories appear in the image, and Inter-scenario mAP that detecting all the categories. The authors provide the benchmarks results on the proposed task with a new OFA-DOD baseline.
Strengths
- The authors analysize the limitation of previous tasks - The authors provide comprehensive benchmark results on the proposed new settings
Weaknesses
First, I want to clarify two different tasks: category-level object detection and visual grounding. The category-level tasks like OVD focus on recognizing pre-defined C categories, e.g., "cat," "dog," etc. These categories are commonly mutually exclusive (COCO) or share a parent-child relationship (LVIS). The OVD task will divide the C categories into base and novel two disjoint groups and the models are required to train on base classes and test on novel classes to verify the generalization ability. Instead of detecting C classes, the grounding tasks like REC, aim to align each region to a text phrase. The REC datasets like RefCOCO do not pre-defined a set of categories. Based on my personal understanding on the OVD and REC tasks, I think this paper has the following weaknesses: 1. The authors argue that DOD is a superset of OVD (#35 and the right part of Figure 1). But I disagree with this argument. The OVD task focuses on evaluating the generalization ability. However, for the DOD task, the proposed dataset evaluation metrics do not consider dividing base/novel groups. Thus, OVD is suitable to evaluate the generalization ability, but DOD is not suitable. If the authors think that the DOD task does not focus on the generalization ability, they should not argue that "superset of OVD". - 1.1 For the proposed D3 datasets for DOD, the authors pre-defined 422 categories. What happens to the other expressions that are not pre-defined? The authors may add some novel classes in the test set that are not available during training to simulate this situation. I think the base/novel setting is one advantage of OVD over DOD. If the proposed D3 dataset has already some novel expression in the test set, the authors may consider adding evaluation results on base/novel groups. - 1.2 The left part of Figure 1 is the definition of object detection rather than open-vocabulary object detection with the base/novel setting. - 1.3 From my perspective DOD is focused on fine-grained language understanding (e.g., the unrestricted language description in #123-127), which is not the main focus of OVD. 2. #188-122: For the REC task, the model is required to align the region with the input language expression and the datasets of the REC task. In my opinion, I don't consider "complete annotation" to be a disadvantage of REC. For instance, REC datasets such as RefCOCO & RefCOCO+ consist of 141,564 language queries, which is significantly larger than the proposed D3 dataset with only 422 language queries. Can we conclude that the vocabulary size of REC datasets is larger than that of DOD? Can we argue that due to the cost of annotation, the "complete annotation" requirement restricts the vocabulary size?
Questions
3. In the proposed DOD task, the category-level definition is vague, and let me feel confused. The authors pre-defined 422 expressions or categories and use these 422 categories for evaluating the Inter-scenario mAP. But are these categories mutually exclusive or overlapped? For example, I guess some short expressions (e.g., 'backpack') may be a parent node of other long expressions (e.g., 'backpack with yellow color'). The authors may consider this point when designing their evaluation metrics, like adopting the positive/negative labels in the OpenImage and LVIS datasets. - 3.1 Figure 3 (b) and (c) use the "number of instances", does it mean "number of categories"? 4. The motivation for the new DOD task is not clear to me. Can the authors provide straightforward application scenarios or real-world examples that previous tasks are limited while the proposed new DOD task offers broader possibilities?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
In #341-345, the authors have discussed their limitations regarding potential abuse and carbon emissions.
Summary
This paper introduces a new Described Object Detection (DOD) task, which extends the existing Open-Vocabulary Object Detection (OVD) and Referring Expression Comprehension (REC) tasks into a more general paradigm. For this new task, the authors build a Description Detection Dataset (D3), and find the troublemakers that currently hinder current REC, OVD, and bi-functional methods. They further propose a baseline that outperforms existing methods on the DOD task.
Strengths
1. Focusing on language-driven object detection, the authors propose a new DOD task, which they argue is more practical and presents more challenges than the current OVD and REC tasks. They also introduce a new dataset D3 for this task. 2. On the DOD task, the authors thoroughly investigate the challenges faced by current REC, OVD, and bi-functional approaches, and they put forth a baseline method with state-of-the-art performance on this task.
Weaknesses
1. While the authors introduce this new DOD task, instead of constructing a new training dataset or setting, they focus on analyzing the performance of models trained on the old OVD or REC tasks for this new task. This may not fairly represent the performance of existing methods for the new task. 2. As the DOD task can be seen as a more generalized version of the OVD task, where category names are extended to language descriptions, the experimental results and conclusions for OVD methods highly depend on their training data and settings. It would be beneficial to assess OVD baselines trained following the style and setting of the DOD task. 3. The compatibility of the DOD task with the existing OVD and REC tasks is not studied. Given that the DOD task is essentially a superset of the OVD and REC tasks, it would be beneficial to evaluate the baseline OFA-DOD in comparison with other methods on the OVD and REC tasks. 4. The existing methods and the proposed baseline are evaluated and compared under a zero-shot setting. How would the performance vary when the models are fine-tuned on the D3 dataset?
Questions
Please refer to the weakness part.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors have discussed the limitations and broader impacts.
Summary
Brief Summary: The paper presents a new task Described Object Detection which extends open vocabulary object detection (OVD) to use phrases. This, in turn, extends referring expression (REC) to include objects not seen in the training data. To this end, the authors introduce a new dataset called D3 building on existing GRD dataset [44]. The key idea behind creating D3 are to have complete annotation (i.e. each referring expression has a bounding box in each image if present), have natural language (extends OVD), include absence expression (i.e. objects NOT having a particular feature / attribute such as blackboard with no signs), and one expression referring to more than one instance. The authors experiment on the provided D3 dataset and provide a detailed benchmark with multiple baselines ranging from REC methods like OFA, OVD methods like OWL-ViT and bifunctional method like Grounding-DINO. The authors further propose a new baseline OFA-DOD which changes some pre-training schemes such as including additional localization tasks and find that it outperforms competitive baselines.
Strengths
Pros: 1. Dataset contribution is always welcome. It is clear the authors have put thought into the dataset construction. In particular, absence expression are quite interesting. 2. A number of baselines are considered and the takeaways are quite interesting that REC methods fail at this task due to being unable to effectively choose more than one boxes. While OVD methods perform better, there is a very large gap compared to bifunctional methods like grounding-dino. The proposed baseline OFA-DOD makes sense and it is good to see that it outperforms other baselines. 3. Visualization of the dataset as well as results in suppl. are very useful. Ablative studies on the baseline such as effect of training data (Table 5b) are interesting.
Weaknesses
Cons: 1. The authors should compare their work with zero-shot grounding [Ref1] which also extends REC to new objects. The obvious difference is that DOD can have more than one instances but a clear distinction would be helpful. 2. The main idea behind DOD is to encompass both OVD and REC. The authors need to motivate this setting more. In my opinion, having the two cases separate can be much more revealing than trying to combine the two. OVD is strictly object detection (with phrases) while REC explicitly requires disambiguation between different objects. For instance, "oversized glove on left-hand" in suppl fig 2 (last row), is simply object detection of glove and doesn't require reasoning whether it is "oversized" or on "left-hand". To me a more natural setting is to separate the two. I would like to know about the author's motivation for the task. 3. It is unclear in the text but in OFA-DOD which OFA is chosen, base or large? To have a fair comparison with Grounding-DINO it should be base but it is not clear (in main text as well as in suppl). 4. In suppl. Table 2, it seems Grounding-DINO outperforms OFA-DOD by considerable margin on Average Recall. Why is this the case? For practical use, wouldn't one prefer using Grounding-DINO? [Ref1]: Sadhu, Arka, Kan Chen, and Ram Nevatia. "Zero-shot grounding of objects from natural language queries." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4694-4703. 2019.
Questions
See weakness section.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes, the authors have discussed some limitations but more could be included such as: 1. The evaluation is heavily dependent on the choice of 412 phrases and the dataset used for GRD.
Summary
The paper proposes a task called described object detection which involves detecting objects through free form text queries, encompassing referring expression comprehension as well as open vocabulary detection.
Strengths
## Clarity The paper is written quite clearly. ## Originality, Quality and Significance: Please see weaknesses
Weaknesses
# Major issues: ## Quality [Annotating using CLIP is insufficient] * DOD does not provide manually annotated explicit negative certificates for the images that are deemed as a negative for a given text query. The negatives are extracted using a CLIP matching score for each image with all the possible text queries. It has been demonstrated in several works that CLIP has no fine-grained understanding of the image [1,2,3], is largely incapable of performing spatial reasoning off-the-shelf, obtaining chance performance even on simple synthetic images [1] and also behaves as a bag of words when understanding the text [2]. This implies that any complex query that is more than just a category name, cannot be appropriately distinguished as being relevant or not for a given image, and using this as a step in the annotation pipeline is sure to introduce errors. Using any sort of image-text model during the annotation process inherits the biases of the underlying model and in my opinion is not a viable approach for constructing an evaluation benchmark. For a benchmark that is characterized as a detection dataset, having accurate negatives is paramount, and the authors have not demonstrated that this criteria can be met using a CLIP matching in the annotation process. * Further, testing the ability of models to distinguish the presence or absence of a textual query, and localize it, would be truly tested in the cases of having unlikely phrases or combinations of objects and attributes or unlikely relations. Using CLIP in the annotation process would completely fail in these cases, as it has been shown that CLIP has a strong Concept Association Bias [3], frequently giving the highest matching score to the most likely completion, without paying attention to the image. This would further exacerbate the difficulty in accurately evaluating models on examples that might be especially hard and interesting (especially relevant to the more challenging "absence" type of queries). ## Originality, Significance [Proposed task is equivalent to existing benchmarks] * As far as I can tell, the proposed task does not differ from the Phrase Detection task proposed by [5] which addresses the problem of both identifying whether the phrase is relevant to an image and also localizing the phrase, across a whole dataset. Departing from referring expression comprehension, they allow prediction of multiple boxes per phrase, and different from object detection, they evaluate text prompts that are longer than simple category names. Missing this line of literature completely is quite a red flag given that it has been around for quite some years now. * Another benchmark, "COPS-Ref" has also been proposed [6] that focuses on referring expressions with varying degrees of complexity and in which the localization must be done across multiple images, also containing distractors. This work also does not acknowledge or differentiate from COPS-Ref. [1] ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension. Sanjay Subramania et al, 2022 [2] When and why vision-language models behave like bags-of-words, and what to do about it? Mert Yuksekgonul et al, 2023. [3] When are Lemons Purple? The Concept Association Bias of CLIP. Yutaro Yamada et al, 2022 [4] Revisiting Image-Language Networks for Open-ended Phrase Detection. Plummer et al, 2020 [5] Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension. Zhenfang Chen, 2020
Questions
## Suggestions * Line 103: "Currently, OFA holds the SOTA among REC methods." I believe this comment is a quite outdated and can be updated (joint OVD & REC methods such as FIBER [1] outperform it). * For the rebuttal, the authors could explain the difference between the proposed task and phrase detection. [1] Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone. Dou et al, 2022
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
1 poor
Presentation
3 good
Contribution
1 poor
Limitations
Yes
Response to rebuttal
Thanks for the clarifications and for the detailed response to my questions. I have one follow up question after which I would be happy to raise my score. Reviewer YHXi brought up the point about evaluation on phrases that may be contained within a longer phrase (such as "backpack" and "yellow backpack"). Could you please clarify how this is handled by clearly explaining how the mAP metric is calculated in these cases or in the case of hypernyms like boat / canoe?
Response to Reviewer Q5JG' Feedback
Thank you very much for reviewing our responses. We are grateful for the feedback! We address the new query with the following points: 1. **Regarding category relationships, parent-child or synonym are avoided in design, but partial overlap is acceptable**. When designing the categories, we intentionally avoided incorporating such parent-child relationships between categories, and also synonym relationships, to ensure greater diversity and challenge within the dataset. However, there is some partial overlap between categories. For example, "dog not lead by rope outside" and "clothed dog" do not have a parent-child relationship but can overlap in certain cases. 2. **Detection on $D^3$ is multi-label, making it suitable for categories with relationships**. Considering possible relationship between categories, detection on $D^3$ is multi-label rather than single-label. An effective detector should assign all relevant positive categories (e.g., "dog not lead by rope outside" and "clothed dog" for a clothed dog not lead by rope outside) for an instance. 3. Given the multi-label setting, **our exhaustively labeled dataset does not require a specially designed metric for category relationships.** In $D^3$, as all positive and negative labels are known for an instance, the relationships between different categories will not affect the evaluation, so we can use consistent evaluation for each category across all images. Comparatively, for datasets like LVIS with non-exhaustive federated labeling, when relationships between categories exists, the partial labels can introduce errors on unknown categories, so such categories may need to be handled specially. $D^3$ is not susceptible to this issue. 4. **We use the standard detection mAP as the evaluation metric.** The evaluation is similar to COCO mAP and we base its implementation (will be open-sourced) on `pycocotools` . For inference, an instance predicated with category A and B is regarded as an instance for category A and an instance for B. The AP for each category is computed as follows: *Predictions for each category across all images are sorted by score in descending order, and those with a ground truth IoU exceeding a threshold are counted as TP (and the ground truth is marked as taken), while the rest are counted as false positives.* With these TP and FP instances, we calculate the precision, recall, and AP following COCO. The mAP is calculated by *averaging the AP across all categories*. In conclusion, the exhaustive annotation in our dataset, unlike federated datasets such as LVIS, and the multi-label setting, which accommodates categories with relationships, ensures that direct AP evaluation for each category is suitable and does not introduce errors. This is attributed to our dedicated design of dataset metadata and annotation process. The standard mAP metric adopted shows our dataset adheres to the stringent requirements of a standard detection dataset and meets the demands of the DOD task. If the reviewer has any further inquiries, we will be happy to answer. Thanks for your help in making this work better.
Thanks for the details!
This sounds good to me! Thanks for the great new dataset! If I may suggest one final thing - I think this paper would benefit from having a more descriptive title. The current one throws the reader off because it's hard to know what the paper is about (since "exposing the troublemakers" doesn't provide any information, and described object detection is not a standard term as far as I know). Maybe consider changing it to be more descriptive of the dataset you are proposing so it has more impact!
Thanks for your review!
Thank you sincerely for your invaluable feedback and comprehensive response. Your review contributes to the enhancement of this work. We wholeheartedly appreciate your insights and will seamlessly integrate our discussion with the existing materials. In regard to the title of the paper, we are fully aligned with your perspective. Your suggestion resonates with us, and we acknowledge that the current title is not clear and informative enough. We are firmly committed to revising the title to be more descriptive of the proposed dataset.
> Is DOD a superset of OVD or not? 1. Results on *base* classes Before reading the rebuttal, I think the proposed method OFA-DOD is trained on the proposed DOD dataset and evaluated on the DOD dataset, so my concern is about missing *novel* classes for generalization. According to the rebuttal, OFA-DOD is trained on OVD/REC datasets and evaluated on the DOD dataset. So my concern now is about missing the results on *base* classes. For the OVD setting, the evaluation is conducted on both the *base* and *novel* classes. However, the authors only report the performance on *novel* classes (e.g., on their proposed new dataset.) But the performance on *base* classes (e.g., on the Open-vocabulary COCO dataset, which is the training set used for DOD) is not provided. So I think missing the evaluation results on *base* classes is a weakness if the authors still argue that the DOD is a superset of OVD. 2. The authors replied that “D3 is exclusively designed for evaluation, and its categories have little overlap with the categories in the training datasets (OVD, REC, etc.), given that the 422 reference categories in D3 are specifically designed.” The ‘little overlap’ is not a rigorous term in an academic paper. How many classes are overlapped? For long-term expression, is there any synonyms in the proposed new dataset? The authors may provide more details. > Are categories in D3 mutually exclusive or overlapped? Consider this when designing the evaluation metrics. The authors replied that "When designing the dataset's categories, we deliberately avoided including categories with hierarchical relationships (such as "backpack" and "backpack with yellow color") to prevent D3 from becoming too straightforward in terms of difficulty." **I think the hierarchical relationship is common in real-world scenarios**. Many existing datasets have a hierarchical relationship, e.g., the ImageNet / LVIS/ Open-Images datasets. But the authors deliberately avoid the hierarchical relationship, which raises my concerns about the authors arguing that their new setting is a real-world task. > Other feedbacks Some of my questions are not answered: 1. For the proposed D3 datasets for DOD, the authors pre-defined 422 categories. **What happens to the other expressions that are not pre-defined?** 2. REC datasets such as RefCOCO & RefCOCO+ consist of 141,564 language queries, which is significantly larger than the proposed D3 dataset with only 422 language queries. Can we conclude that the vocabulary size of REC datasets is larger than that of DOD? **Can we argue that due to the cost of annotation, the "complete annotation" requirement restricts the vocabulary size?**
Response to Reviewer YHXi's Feedback
Thank you for the responses. ### 1. Missing the results on *base* classes if DOD is a superset of OVD. - **1**. By definition, OVD aims to generalize beyond *base* classes during training, and detect *novel* classes defined by an open vocabulary at inference. The definition of DOD is the same, except the classes are unrestricted references rather than only short class names. $D^3$ qualifies as a DOD evaluation set as it provides *novel* classes. - **2**. **OVD task primarily focuses on evaluating performance on *novel* classes** after training on *base* classes, and evaluation on *base* is mainly an experimental setting to check the capability on seen classes and the upper bounds on unseen. (Some methods, e.g., MEDet and the SOTA CORA, do not provide LVIS *base* performance). The reviewer also mentioned in the original comment that OVD focuses on the generalization on *novel*, and our zero-shot evaluation on $D^3$ also reflects this perspective. - **3**. In our evaluation, different baselines are trained on varying tasks and datasets, making it challenging to assess the performance on their *base* classes. - **4**. In response to the suggestion, we **reconstruct **$D^3$** for training and eval.** We splitted $D^3$ and obtained a training set (*base*: 259 classes) and two test sets (*base*: 259, *novel*: 126). We conducted training on *base* and evaluated on both *base* and *novel*. The results are presented in the table below. | Model | Novel | Base | | --- | --- | --- | | OwlViT | 9.7 | 15.2 | | OFA_base | 4.3 | 11.3 | | UNINEXT_huge | 18.6 | 23.8 | | OFA-DOD | 21.6 | 25.1 | ### 2. Quantify the minimal overlap between *base* and *novel* classes for DOD task on $D^3$. Thanks for the suggestion regarding rigor of the expression. We've analyzed category overlap between *base* (OVD dataset: COCO/LVIS; REC dataset: RefCOCO/+/g) and *novel* ($D^3$), by: - For OVD, we used ChatGPT to generate synonyms from categories, matching them against $D^3$ references. Overlaps: COCO 0.4%, LVIS 0.9%. - For COCO, which have less categories, we also perform manual check, resulting in 0.7% overlap with $D^3$. - For REC, we apply a threshold on the sentence similarity calculated via HuggingFace's `bert-base-cased-finetuned-mrpc` model. Overlaps of $D^3$ with RefCOCO/+/g: 0.0%, 0.2%, 0.7%. Thus, *novel* classes ($D^3$) overlap <1% with *base* classes (OVD & REC). ### 3. Avoiding hierarchical relationship in the dataset may raise concern on if the new setting is real-world task. We argue that avoiding such hierarchy does not affect the real-world attributes. The possible references in real world are infinite and we can only annotate some of them. Considering our restricted annotation capacity, **we forgo hierarchy to concentrate more on intricate categories and make the descriptions more diverse and rich.** Keeping hierarchical refs like "backpack" and "yellow backpack" seems redundant compared to others, given their similarity. Due to annotator limitations, our categories are limited. Thus, we prioritize informative, diverse refs. **Our detection task is multi-label, not single-label, accommodating complex relationships between refs.** We include intricate references like "clothed dog" and "dog not lead by rope outside," with partial overlap and no simple inclusive hierarchy. ### 4. What happens to expressions not pre-defined, except the 422 in the dataset? Sorry but we didn't quite understand the reviewer's question. If the reviewer's inquiry pertains to whether the $D^3$ dataset supports evaluation beyond the 422: $D^3$ does not support evaluation for categories beyond those defined, as the ground truth annotations are exclusively available for these categories. These categories represent a subset of reality, used to assess method performance, similar to detection/REC datasets. No dataset we know so far can cover all possible references in the world. If the reviewer is inquiring about whether the DOD method supports inference for categories beyond the 422 defined: These baselines could theoretically infer from categories outside the 422, but performance might vary significantly, as seen in zero-shot results differences on the 422 categories. ### 5. Is the vocabulary of REC datasets larger than $D^3$ for DOD? Did the need for "complete annotation" limit vocabulary due to annotation costs? Yes to both. Though our positive and negative sample count surpasses REC's test set, REC's vocabulary remains notably larger than DOD's due to the complete annotation requirement of DOD. Enforcing complete annotation, akin to REC's large reference classes, is unfeasible for us due to high costs. Creating our 422-category dataset involved the contributions of a team of 15, spanning nearly 2 months, costing around $11,000. We believe a reference-based dataset with complete annotation can be a valuable community resource. It acts as a starting point, potentially inspiring diverse and comprehensive future DOD datasets.
Thanks to the authors for providing more details like the overlap between the proposed dataset with previous OVD/Det/REC datasets, which I believe will make a more solid manuscript. > Missing the results on base classes if DOD is a superset of OVD. For the OVD dataset, achieving strong results in new classes without compromising performance in base classes is essential. I think it's important to present results for both the base and new splits, as clearly outlined in the OVD setting (for instance, OVR-CNN/ViLD). I appreciate the authors providing more results like reconstructing the proposed dataset for training and eval. > About the hierarchical relationships I think the COCO-style mAP evaluation is not suitable for real-world settings. COCO is defined as 80 classes (human-selected and relatively small) and avoids hierarchical relationships. In recent years, large-scale real-world datasets such as LVIS (with 1,203 classes) and OpenImages (with 500 classes in the v4/v5 version) have demonstrated that as the number of categories increases, challenges related to hierarchy and synonymy arise. This is why both LVIS and OpenImages have introduced new evaluation metrics, such as federated evaluations with positive/negative labels. Such federated evaluations will be also beneficial to avoid the high labeling cost. In fact, previous datasets such as LVIS and OpenImages are also multi-label. It's natural to use federated evaluation for assessing multi-label datasets. However, the COCO-style mAP evaluation doesn't support multi-label evaluation. The dataset proposed, which uses a COCO-style mAP evaluation, implies two main constraints: (1) the number of categories must be limited, and (2) the categories must be predefined or selected by humans. This is why I believe such an evaluation pipeline doesn't reflect a real-world task setting. By contrast, the federated evaluation settings in the style of LVIS/OpenImages accommodate a greater number of categories with more complex relationships.
Response to Reviewer YHXi's Official Comment
Thanks for reviewing our responses. > 1. About the results on *base* classes Thanks for the feedback. We are happy that the reviewer recognized the results we provided on both *base* and *novel*. We will add the discussion and analysis above in the manuscript. Please let us know if there is any clarification needed. > 2. About the hierarchy relationships We summarize the reviewer's opinion in this part as: the reviewer thought the complete annotation and the corresponding COCO-style mAP metric for the dataset is not realistic, and encourages using federated annotation and evaluation, similar to LVIS, which may brings the benefits of more categories, with hierarchy or synonym relationships and not pre-defined. We response to this opinition with the following points: **(1)** We agree that federated annotation & evaluation brings the benefit of more category and more relationships, **but not the benefit of no pre-defined categories**. For expression-based dataset, we still need to pre-define and human-select categories even with federated annotation and evaluation. This is because expressions, unlike object category names, are infinite on a set of images, due to the versality of language, so **annotators always need to select manually some expressions from the infinite possible ones.** If we simply annotate objects in images with expressions human-selected but not pre-defined (i.e., whatever the annotator comes up with), then we will get a dataset with a lot of categories but one or few instances for each category, similar to a REC dataset, not a DOD dataset. Thus, **a expression-based detection dataset like **$D^3$** requires us to pre-define and human-select expression categories.** **(2)** We argue that **federated annotation & eval or complete annotation & COCO-style eval have different advantages and the choice depends on the situation**. Complete annotation & COCO-style eval brings more positive and negative samples annotated and evaluated (from all the images in the dataset) for each category, while federated annotation & eval brings larger number of categories but less samples (from only a subset) for each category. **(3)** **For DOD, currently we prefer complete annotation & eval over federated annotation & eval, because we need more negative samples with explicit negative certificates** to verify a model's ability to distinguish and reject negative instances, as described in the motivation in the [general response](https://openreview.net/forum?id=0hwq2vOHT4¬eId=EtVOyLQxeQ). A dataset with federated annotation & eval is nice for the advantages, but not the original target of this work. **(4)** Our dataset focuses on utilizing completely annotated samples to validate the capabilities to detect (discover, locate and reject) instances with flexible expressions. Currently, there are other potential aspects not taken into account, such as the influence of expression relationships. We will continue to investigate them. It is conceivable that in the future, we may witness the emergence of datasets based on $D^3$ featuring federated labeling and encompassing a broader range of categories. Our work serves as a kick-off point for DOD. We would like to clarify there that the contribution majorly involves a dataset for detection with flexible expressions, accompanied with task setting, comprehensive experimental analysis and a new baseline. We earnestly hope that the esteemed reviewer takes into account the potential contribution of this work. **(5)** One more thing to note: the reviewer mention that _COCO-style mAP evaluation doesn't support multi-label evaluation_. As far as we know, when the annotation is not complete (with federated annotation, like LVIS/OpenImage), it does not support multi-label eval, but **with complete annotation, COCO-style mAP is feasible**. Examples of multi-label evaluation using COCO-style mAP includes multi-label image classification on COCO and Human-Object Interaction Detection on HICO-Det, to name a few. Our annotation is complete so COCO-style mAP is plausible as the evaluation metric.
My argument is that the COCO-style mAP evaluation is not suitable for a real-world setting. That’s because COCO-style mAP evaluation, like this paper, needs disjoint labels, and the number of classes cannot be very large. > (1) We agree that federated annotation & evaluation brings the benefit of more category and more relationships, but not the benefit of no pre-defined categories. For expression-based dataset, we still need to pre-define and human-select categories even with federated annotation and evaluation. This is because expressions, unlike object category names, are infinite on a set of images, due to the versality of language, so annotators always need to select manually some expressions from the infinite possible ones. If we simply annotate objects in images with expressions human-selected but not pre-defined (i.e., whatever the annotator comes up with), then we will get a dataset with a lot of categories but one or few instances for each category, similar to a REC dataset, not a DOD dataset. Thus, a expression-based detection dataset like $D^3$ requires us to pre-define and human-select expression categories. The author may misunderstand my concern. My concern is about pre-defined ‘**disjoint**’ classes. In the real-world scenario, as the number of classes increases, other types of pairwise relationships such as partially overlapping or parent-child will occur. Previous federated evaluations like LVIS/OpenImages support such hierarchy label structures. But this paper uses manual selection to avoid complicated relationships, so I don’t think it’s a real-world setting. The authors mention that “annotators always need to select manually some expressions from the infinite possible ones”. In fact, this is just one advantage of the federated setting to reduced workload. Federated datasets do not need exhaustive annotations. > (2) We argue that federated annotation & eval or complete annotation & COCO-style eval have different advantages and the choice depends on the situation. Complete annotation & COCO-style eval brings more positive and negative samples annotated and evaluated (from all the images in the dataset) for each category, while federated annotation & eval brings larger number of categories but less samples (from only a subset) for each category. I think the latter (larger number of categories but fewer samples) is more like a real-world setting. In fact, as the number of categories increases, some of the frequent classes will have sufficient samples, while some of the rare classes will have insufficient samples, like the LVIS/OpenImages dataset. Previous open-vocabulary object detection tasks also use LVIS as their benchmark for real-world consideration. > (3) For DOD, currently we prefer complete annotation & eval over federated annotation & eval, because we need more negative samples with explicit negative certificates to verify a model's ability to distinguish and reject negative instances, as described in the motivation in the general response. A dataset with federated annotation & eval is nice for the advantages, but not the original target of this work. The federated evaluation also measures the model’s ability to reject negative instances. You may refer to section 2.2 in the LVIS paper. > (4) Our dataset focuses on utilizing completely annotated samples to validate the capabilities to detect (discover, locate and reject) instances with flexible expressions. Currently, there are other potential aspects not taken into account, such as the influence of expression relationships. We will continue to investigate them. My concern is that the current evaluation pipeline is somehow not real-world with many restrictions. Glad to see that the authors will consider investigating more complex relationships. I think the federated evaluation is a good choice, which has been demonstrated in previous LVIS/OpenImages papers. > (5) One more thing to note: the reviewer mention that COCO-style mAP evaluation doesn't support multi-label evaluation. As far as we know, when the annotation is not complete (with federated annotation, like LVIS/OpenImage), it does not support multi-label eval, but with complete annotation, COCO-style mAP is OK. Examples of multi-label evaluation using COCO-style mAP includes multi-label image classification on COCO and Human-Object Interaction Detection on HICO-Det, to name a few. Our annotation is complete so COCO-style mAP is plausible as the evaluation metric. First, I want to argue that federated evaluations like LVIS/OpenImage support multi-label evaluation. You may refer to Figure 2 and Section 2.2 of the LVIS paper. Some of the datasets build on MS-COCO support multi-label classification, but they use the federated evaluation setting. For example, the mentioned case, HICO-Det, is also a federated evaluation setting with positive/negative groups.
Response to Reviewer YHXi's Recent Comment
Thanks for the detailed responses. We greatly respect and appreciate these. > COCO-style mAP eval is not suitable for real-world setting, because COCO-style mAP, like this paper, needs disjoint labels, and class number cannot be very large. For COCO-style mAP, we agree that *class number cannot be very large* for annotation cost. As $D^3$ is for eval, we do not aim to develop a dataset with thousands or more classes in this work, and 422 is a balance between dataset scale and cost. We prefer complete annotation with more positive and negative samples as this is the first for DOD. We want to argue against *COCO-style mAP eval needs disjoint label*: since we use multi-label classification setting, classes with overlap are OK as long as completely labeled like $D^3$ (and many others, like HICO-Det, which has non-disjoint labels but adopts COCO-style mAP and complete annotation). Our dataset provides classes like "clothed dog" and "dog not led by rope outside", which are partial-overlap & non-disjoint but OK for eval, as they are completely labeled. > ... My concern is about pre-defined ‘disjoint’ classes. In real-world scenario, as class number increases, other pairwise relationships such as partially overlapping or parent-child will occur. Federated evals like LVIS support such hierarchy label structures. But this paper uses manual selection to avoid complicated relationships, so I don’t think it’s real-world setting. About "disjoint": as in the paragraph above, though we avoided hierarchy (simply to make the classes richer and more diverse), there are other non-disjoint labels like those with partial overlapping. About "pre-defined" "manual selection": As in the last response, manual selection can be avoided for object name-based detection like LVIS as object names in a dataset are limited to several thousands. But for expressions which can be infinite, there are always manual selection needed. > The authors mention that “annotators always need to select manually some expressions from the infinite possible ones”. In fact, this is just one advantage of federated setting to reduce workload. Federated datasets don't need exhaustive annotations. The reviewer seems to agree "annotators always need to select manually some expressions from the infinite possible ones", which explains manual selection is necessary for expression-based dataset like ours. We agree that federated datasets don't need exhaustive annotation. > I think larger class number but fewer samples is more like real-world setting. As class number increases, some frequent classes will have sufficient samples, while some rare classes will have insufficient ones, like LVIS. ... We agree that federated annotation is good for larger class number. We will consider using federated annotation if we are to build a dataset with larger class number based on $D^3$. Currently it serves as an eval benchmark and a kick-off for DOD. Thanks for pointing this direction. > The federated eval also measures the model’s ability to reject negative. ... We did not suggest that federated eval doesn't measure the model's ability to reject negative. We were claiming complete annotation and eval, compared with federated, provides **more** positive and negative samples and focuses **more** on evaluating that ability. As the first for DOD, we are not able to cover all aspects of an ideal dataset, and focus less on large class number. A federated dataset may be a future work. > My concern is that current eval pipeline is not real-world with many restrictions. Glad to see that the authors will consider investigating more complex relationships. ... Thanks for the comments. All your suggestions are helpful to us. > First, I want to argue that federated eval like LVIS support multi-label eval. ... Some datasets build on MS-COCO support multi-label, but they use federated eval. For example, the mentioned HICO-Det is also federated eval setting with pos/neg groups. We did not suggest federated eval doesn't support multi-label. What we tried to clarify is **COCO-style mAP is OK for multi-label setting**, for this was queried by the reviewer previously. Both HICO-Det for HOI and COCO for multi-label image classification adopts complete annotation & COCO-style mAP, rather than federated eval. If the reviewer has questions regarding their eval, references can be found in HICO-Det paper (WACV18) like "For each HOI category, we evaluate detection on the full test set" and "we evaluate HOI detection using mAP", and in the `compute_map` functions of the official implementation of various methods, like QPIC (CVPR21) and PPDM (CVPR20); the mAP for COCO multi-label image classification are available in the `mAP` class in the implementation of ASL (ICCV21). We mentioned these to prove COCO-style mAP is feasible for multi-label. These discussion are very helpful to this work and we would like to express our gratitude to the reviewer for their detailed comments.
Thank the authors for their detailed response. I appreciate their efforts in addressing my concerns. However, I still believe that having a proper training dataset is more critical and necessary for this DOD task and could greatly enhance the impact of this work. Using existing OVD/REC data to train DOD models may not fully exploit the true potential of current methods or models. Therefore, I would like to maintain my original rating.
I’ve been following this discussion and would like to share my thoughts - given that the paper has quite clearly mentioned that it is purely an evaluation benchmark (with 100k images no less), I don’t think it’s fair to evaluate it on the basis of not having a training set. There are plenty of papers (such as Winoground, SVO-Probes, etc) that have been proposed to uncover the limitations of current VL models and have been quite valuable to the community. As you mention « Using existing OVD/REC data to train DOD models may not fully exploit the true potential of current methods or models. » , I believe this is exactly the thing this benchmark aims to measure, ie the fact that there is no good model that can do both REC and OVD. Overall, I think it’s a good addition to the set of available evaluation benchmarks.
Reply to Rebuttal
After having read the rebuttal to my questions, as well as the replies to the other reviewer's questions, I have no further questions at this point and I will stand by my current evaluation score.
Thank you for the response. Additional details on the annotation process are helpful, as well as the added motivation for the task. The visualizations provided in the paper however don't have "oversized" glove, and my example was based on that example. It is unclear how many other such examples are there and perhaps more qualitative analysis could be useful. As such, I keep my score.
Response to Reviewer SdJt's Feedback
Thanks for the feedback. The reviewer questioned the annotation of "oversized glove on left hand" (last row) in Fig. 2 of supplementary material is indeed annotation on "glove". We want to clarify that the "oversized glove" refers to gloves significantly larger than regular size (i.e. human hand size). As an evidence for this, in the rightmost image of this row, there are two gloves with regular size on both hands of the baseball player on the left. As these two gloves are **not "oversized" and "on left hand" at the same time, they are not annotated. This shows we are actually annotating "oversized glove on left hand" rather than "glove".** Gloves not significantly larger than regular size or gloves on right hand were not annotated. In this figure we mainly shows the "unrestricted descriptions" characteristic of the dataset, so we use mostly positively annotated samples of "oversized glove on left hand", which are also positive for "glove". As a better example, we shows some samples of cars damaged (annotated) and not damaged (not annotated) for "partially damaged car" category in the examples in Fig. 1 of supplementary material, which focuses on "complete annotation" (of both positive and negative samples) for the dataset. We hope this would clarify the reviewer's question on annotation and we would add more qualitative cases, along with discussions, in the manuscript. Thanks for the suggestions.
Decision
Accept (poster)