Summary
The authors propose a prototypically-aligned contrastive learning framework for vision and language in order to enable better web-scraping of fine-grained, rare concepts that are easily confused or mapped to other more common concepts when either vision or language is considered in isolation (what they term “semantic noise”). Their goal is to improve webly-supervised learning, which is often plagued by label noise, particularly for fine-grained categories. They call their method CAPro, for “Cross-modality Aligned Prototypes.” Their method is built from a few simple ideas: 1) use text prototypes, well-defined in the literature, to scrape a cleaner set of visual prototypes for fine-grained classes, 2) Use visual features to fill gaps or correct errors within text prototypes for fine-grained concepts, generating “semantically-aligned visual prototypes,” 3) an additional cleaning step to further reduce noise between the visual and textual prototypes that uses cluster regularization, 4) “collective bootstrapping” to further smooth concepts or “label references” via essentially an adapted/aggregated/bootstrapped dictionary lookup over the entire dynamic concept dictionary that reduces the effect of examples divergent from the average when making predictions, particularly helpful for visually similar classes. Their method is highly similar to MoPro, but with the addition of the above described semantic noise correction and regularization across modalities.
After reading the authors rebuttal and discussing with them, as well as reading the comments and discussion with other reviewers, I will increase my score to a 6.
Strengths
The authors show that their method performs well for single-label and multi-labe, on WebVision1k and Imagenet1k, and shows some open-set generalizability. In particular, they define 235 categories from these datasets as exhibiting “polysemy concepts”, and show that their method is particularly effective for these concepts. The gains shown are nice, and the more detailed performance breakdown of the 235 class versions of both datasets is interesting (though it would be nice to also show the performance breakdown for the non-polysemy concepts to help build reader intuition of the impact of polysemy). The method does seem to clearly have an advantage in multi-label challenges, and performs better than CoTeach or VSGraph on open-set concepts. The qualitative examples in table 3 are quite compelling, though they seem potentially highly cherry-picked.
Weaknesses
Notably, their system is quite complex, and feels a bit like a bag of tricks (albeit an effective one), and the method both performs quite similarly to MoPro and often underperforms VSGraph on Top1 (the authors point out that these may not be directly comparable, as they use different backbones, but this makes it difficult to understand or interpret the table. Perhaps they could somehow point out which methods are comparable along which axes in the table? As it is, it’s not very easy to determine the methods value vs other methods). I would also have liked to see more thorough ablations to better understand how impactful their suggested components were vs the additional computational complexity added, to better understand how “worth it” each component would be to use or implement. I find table 4 to be a bit difficult to interpret. I would also like a more thorough analysis of why top-1 suffers, but top-5 benefits (this may be related to over-regularization somehow?).
Nits:
In the abstract: “exacerbates fine-grained visual concept learning.” should perhaps be “exacerbates the challenge of fine-grained visual concept learning.”
Line 26: “The large scale” -> “Large scale”
Line 305: “study” -> “studies”
Questions
I’m curious how this method would handle visual dimorphism in fine-grained classes, for instance dark-eyed juncos which have multiple distinct color morphs, some more common than others. Since the method explicitly seeks prototypical examples, is this at the expense of recognition of anomalies or rarities within-concept? Have the authors seen any interesting failure modes for their method, where simpler methods succeed?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have not addressed this at all, which I see as a significant weakness. What are the potential benefits or harms when building prototypical definitions of concepts? What might this mean for instances which do not fit the prototypical visual appearance of a given concept on the internet? How might biases in internet data limit our prototypes?