DesCo: Learning Object Recognition with Rich Language Descriptions

Summary

The paper describes an approach to open world object detection that generates detailed object descriptions with GPT to improve performance. The main advance of the paper is showing that the removal of the entity name, and just using it's description, forces the network to learn the "context". This is turn leads to better performance on zero-shot tasks. Results are shown on the LVIS and OmniLabel datasets in comparison to GLIP and FIBER.

Strengths

The paper clearly demonstrated that the additional contextual information needed to be presented to the network properly to see a performance boost. The authors' results are clearly better than FIBER and CLIP.

Weaknesses

The writing of the paper make it difficult to understand the approach. I had to read it multiple times to understand how the models were trained and tested. It's still not 100% clear how all the diverse datasets used for training where converted to description-rich data. The paper states "As illustrated in Section 3, we convert the text caption of each instance into a detailed language description to construct description-rich data.". This should be made more concrete. The paper doesn't describe the training of CLIP and FIBER. A little more background would help better differentiate the contributions of this paper. The main novelty of the paper is removing the entity name from the description to force the network to use the contextual description. It is not clear this is of sufficient novelty/interest for NeurIPS. The authors don't explore other prompts for generating object descriptions. This would have greatly added to the paper.

Questions

Where the COCO images removed from training, similar to CLIP? When generating hard negatives (Line 194), how can you ensure they're actually negative? The example of the mallet in the paper states a mallet has a wooden handle, but the image contains a mallet with a red handle (could be made from any material). Is the model really learning a grounding for the context, or just querying with a bag of words? On line 176, it's stated the "mutual information still stays zero". Is this true? The mutual information may be low, but is it really zero?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The paper does not provide potential negative societal impact of their work.

Summary

This paper is about open-vocabulary object detection, in which a model is given an image and a sequence of object descriptions and has to predict bounding boxes matching the descriptions. While open-vocabulary detectors can in principle take arbitrarily detailed object descriptions as input, they tend to focus on the main noun in the description and ignore further details in the description. This is problematic at inference time, where the model might be prompted with descriptions of objects that it has not seen during training, and where leveraging additional detail in the description becomes necessary to solve the task. The present paper addresses this issue by augmenting the training data with detailed object descriptions and then forces the model to use these descriptions. First, standard object detection datasets are augmented with detailed object descriptions by querying a large language model (GPT-3) to provide additional details about object categories occurring in the detection datasets. The language models are also used to generate hard negative descriptions for similar object categories. Second, open-vocabulary object detectors are trained using these descriptions. By combining positives with easily confusable negative descriptions, the model is forced to attend to the details in the description. A crucial detail is that the description should not explicitly name the target object type, such that the model is forced to use the description to solve the task. Using two off-the-shelf open-vocabulary object detectors (GLIP and FIBER), the experiments show that the proposed data augmentation improves performance on the LVIS and OmniLabel benchmarks. Ablations show that the biggest boost in performance comes from adding detailed descriptions that do not explicitly name the target entity to the training data. Hard negatives provide a small additional boost.

Strengths

1. Natural language supervision is an important trend. Therefore, general ideas for how to clean or augment natural language to provide better supervision are valuable. The idea to remove the entity name from descriptions goes in that direction. 2. The analysis of how detection performance is affected by the LLM used for data generation (Table 4) is significant because it shows that language quality is an important determinant for downstream task performance. This is useful guidance for other works using LLMs for data generation.

Weaknesses

1. The contributions are insufficient for a NeurIPS paper. The primary contribution is to apply the Winograd format to query construction for object detection. While it is useful to know that this helps in the case of object detection, the idea has been employed extensively in related contexts. Further, using an LLM to generate the object descriptions is not well motivated and not necessary at the scale explored in the paper. Datasets of object descriptions of the size used here (~10000 entities) already exist; it’s not necessary to use an LLM for that. There are many interesting questions that could be explored to make the contributions more substantial. For example, does performance improve when scaling up to larger numbers of object descriptions, which is possible with LLMs? Does the LLM prompt matter? Could similar results be achieved with existing datasets by simply masking the entity name? 2. The paper lacks detail in the methods and experiments. The proposed approaches could not be reproduced from the given descriptions. For example, much more detail is needed about the generation of negative captions. The paper briefly mentions that negative captions are generated using a multi-stage process that involves first prompting a language model, then training a second language model to generate more data. But no details are given about how the first LLM is prompted or how the second one is trained. Please describe this process in sufficient detail to reproduce it. 3. The paper makes some misleading claims. For example, L279 claims that the proposed method “surpasses supervised models by a substantial margin”. The models labeled as “supervised” in Table 1 are very far from the state of the art of supervised LVIS detectors, which reach around 60 AP and 50 APr. This is therefore a meaningless and misleading comparison.

Questions

Questions 1. LVIS evaluation (Appendix B.3): What exactly do you mean by “fixed AP protocol”? Do you mean the evaluation proposed in https://arxiv.org/abs/2102.01066? That paper makes various recommendations and doesn’t specify a precise protocol. How exactly do you modify the standard LVIS AP metric to “fix” it? Please provide code or a detailed descriptions sufficient for reproduction. 2. Suggestion for improving the paper: Build on the unique possibilities afforded by LLMs and study those more. Scale up, study prompt choices, study description diversity etc. Typos * Typo in abstract: OminiLabel. * L92 missing word: “…pretrains models large-scale image-caption pairs to learn transferrable representations.” * Appendix L37 typo: “protocal”

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The paper does not discuss limitations.

Summary

This paper leverages rich descriptions generated by LLMs to encourage GLIP/FIBER to pay more attention to object attributes aside from object names when performing detection tasks. A more robust model is resulted from training on object-description alignment as the model is able to leverage attributes for recognition as well as avoiding confusable objects. Experiments show that key to the success of this method is the inclusion of hard negative descriptions without object names, in order to effectively motivate the model to use the information in addition to object names. By doing so, the model also overcomes one of the previous limitations where a detection model is always predicting objects even though the description mentions non-existing objects in the given image.

Strengths

1. This paper leverages hard negatives to force a VL model to learn the object-attributes alignment in addition to merely mastering object-noun alignment. This brings significant improvement in zero-shot recognition of long-tail objects. 2. This paper enriches captions in existing image-text datasets with large language models. Those enriched text descriptions can be used to train better language-aware vision models. 3. Overall, this paper demonstrates a viable path of distilling the knowledge present in LLMs into better object grounding abilities in VL models.

Weaknesses

1. There lacks a test for the false-positive rate. The proposed training setup with carefully hard negatives is alleged to address the issue mentioned in line 181-184. However, there lacks evaluation on how much improvement there is with respect to hallucinating entities that are mentioned by the description but don't exist in the image. 2. There lacks a test for how well the model is able to "reject" a grounding attempt based on misaligned attributes. For example, when the image contains a red umbrella, Line 195-196 ("Note that the label of the word mallet is affacted by the context") suggests that the expected behavior is to ground "red umbrella" while not predicting any grounding if the text says "yellow umbrella", even though the entitiy name is correct. It'd be helpful to evaluate how well the model captures the nuances exemplified by these two cases. 3. Lack of error analysis. For example, in what cases do the relative confidence scores for Q+ and Q- flip after Desco training? 4. As shown in Figure 3, although the descriptions might contain multiple entity names, it seems that the target entity is always the first token, while entities appearing later in the description are supposed to be "modifiers". Would this bring an undesirable bias? How well does the model perform if the target entity appears in a different location in the description?

Questions

In Table 3, what is APd-P as apposed to APd?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

More experiments and analyses could be done to better justify the proposed training method.

Summary

The paper tackles the problem of open-world (language-based) object detection and proposes two main things. First, it uses an LLM to obtain detailed descriptions of the object classes of interests and, secondly, proposes a training methodology that forces the network to pay attention to such descriptions through the use of (language) hard negatives.

Strengths

Good paper overall, ticks all boxes. It uses some convincing advanced baselines in a very interesting and current setting, identifies shortcomings, proposes a reasonable solution, and shows the effectiveness well.

Weaknesses

Nothing important to say. As a minor comment, it is a bit hard to be sure about the setting the paper is tackling. The title talks about "object recognition", but the problem is object detection. Then it mentions grounding too. And it is not clear where the bounding boxes come from. Are they proposed by the network? are they given? (one could go to the glip paper, but better be self-contained).

Questions

I have no qualms with the paper as is. A couple minor things the authors might (or might not) find useful: - I like the results split for "rare" vs "frequent" objects, I think it is quite informative, but it is just mentioned in passing. Adding this split in a table and be more complete might be useful? More generally, some insight into what type of categories benefit the most (as in, finer-grained? ambiguous?...) and which ones don't would improve the paper. - the hard negatives technique and the explanations are a hot topic so there's several recent papers. I won't add many arxiv papers, but here's two the authors might find interesting, exploring somewhat similar strategies (though for different problems): CVPR'23 - Teaching Structured Vision & Language Concepts to Vision & Language Models from arxiv: Waffling around for Performance: Visual Classification with Random Words and Broad Concepts

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

This work revisits a key aspect of open-vocabulary object detection: what makes for the class names? It shows that GLIP, the state-of-the-art vision-language model for open vocabulary object detection, behaves like bag of words and could not effectively understand text descriptions of the objects to detect. To tackle this challenge, it turns to LLMs to extend the class names to detailed descriptions to solve the label scarcity issue. It also creates hard negative descriptions to ensure the model focuses on the context. Experiments on two challenging benchmarks (LVIS & OminiLabel) shows strong results and validates the effectiveness of this work.

Strengths

*Originality*: Though the high-level idea of extending the understanding of class names with LLMs shares similarity with [a], the application of this work is fairly original in this field. *Quality*: This work is well-motivated via re-evaluation of GLIP. The idea is practically simple and clear, and its effectiveness is well-supported by strong performance on two challenging benchmarks. Overall it is insightful and well-shaped for acceptance. *Clarity*: The delivery is fairly smooth and clear, and I find no major issues in understanding this paper. *Significance*: Recent works in open-vocabulary object detection still focus on mining region-word (noun) correspondence. I believe the message/intuition from this paper could be helpful for the community. *Reproducibility*: Code & data release could be very helpful. Release upon acceptance is promised. [a] Menon & Vondrick, Visual Classification via Description from Large Language Models, ICLR 2023.

Weaknesses

I did not find major concerns in this work. See next section for questions.

Questions

- More examples, especially failure cases of the LLM-generated descriptions could better help understanding the model. - Currently, the text seems to be more like a general description in terms of representative visual attributes (low-level) and semantic/meaning of the object (higher-level). One interesting direction is to create more diverse descriptions in different aspects (lower/higher-level), and diagnosis the contribution of each. Another question is how well could such fixed text descriptions cope with diverse (or OOD) visual appearances.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

limitations are discussed in the last section

response to the authors

My opinion was already positive to begin with. I've read the other reviews and the rebuttal and there isn't much to change my mind one way or another. I think the paper should be accepted. It seems useful and reasonable, with good results.