Summary
In this paper, the authors more a controlled comparison between two pretraining approaches to learning visual representations from language supervision:
contrastive (CLIP-style) and generative (= image captioning, e.g. VirTex, SimVLM, BLIP, CoCa), etc.
The goal of this paper is not to advance the state-of-the-art, but rather to observe the model/data scaling behavior of these pretraining tasks.
Experiments show that captioning-pretrained models can match or outperform CLIP-style contrastive models on several multi-modal tasks.
Strengths
**I think this paper, in its current form, already matches the quality of a typical publication at the NeurIPS conference.**
I strongly recommend acceptance; it is relevant to the conference audience and will spur interesting discussion in the community.
Below I highlight the main strengths of the paper to substantially support my assessment:
1. **Paper presents contrary evidence to existing results:**
The vision community is making rapid progress with vision-language models as they enable new transfer applications that can be specified using natural language.
Much of this progress in the last 2-3 years was catalyzed by the development of CLIP (and concurrent works like ALIGN).
Ever since, the vision community has largely gravitated towards pushing progress on multi-modal contrastive models,
following the "image captioning models converge slower on web data" result from the CLIP paper.
This paper presents a piece of contrary evidence that simple captioning-only models can match or outperform their contrastive counterparts.
2. **Promising alternative to poor language understanding of contrastive models:**
Image captioning as a pretraining task has been studied in previous models (e.g. VirTex, SimVLM, BLIP, CoCa, etc.).
The main novelty of this work is a direct comparison of captioning with the contrastive objective at scale,
with controlled model capacity and training dataset size.
Hence, I think this is a timely contribution that will force practitioners and researchers to rethink the relevance of text-generative models
and side-step the embarrassing failures of contrastive models, e.g. their inability to distinguish "man eating a sandwich" from "sandwich eating a man".
3. **New evaluations with captioning models show practical runtime solutions for classification/retrieval:**
Captioning models are known for their slow image/text retrieval runtime since they cannot "cache" the text classifier weights once like contrastive models.
However, they are better at language understanding than contrastive models which are known to behave as bag-of-words models.
Authors show evaluations related to "LiT tuning" to convert a captioning-pretrained image encoder to a contrastive model --
this helps overcome the runtime overhead of image captioning-only models.
4. **Experiments are thorough and well presented:**
The paper studies a targeted comparison between captioning and contrastive pretraining approaches.
The authors present a series of experiments and evaluations to support this study.
All comparisons seem fair and controlled to the best of my knowledge, with differences specified wherever relevant.
Modeling ablations and experiments with a different dataset (LAION-400M) make the study more self-contained.
Many evaluations report error bars wherever appropriate.
5. **Excellent clarity in writing and presentation:**
The motivation for this study is precisely stated I the abstract and introduction.
The coverage of related work is broad and comprehensive.
All technical details for empirical analysis are well-stated and easy to follow.
The main paper and supplementary material have adequate implementation details to aid reproducibility.
Weaknesses
I have some questions and suggestions that could make the study more comprehensive.
Have the authors considered the following experiments in their study?
1. **The exact autoregressive language model used in CLIP paper:**
CLIP paper uses a different autoregressive model, and this particular model is shown to converge slowly.
Have the authors tried this exact architecture?
Instead of a transformer decoder with cross-attention to image features (e.g. like VirTex),
CLIP's autoregressive baseline has a SimVLM-style design wherein image features are pooled into 2x2 grid and passed to the text model as the first four tokens.
The model follows the transformer encoder design and predicts caption tokens autoregressively.
2. **Backward captioning or masked language modeling?**
Have the authors considered auxiliary objectives used by prior works, such as backward captioning (VirTex) or masked language modeling (ICMLM)?
These objectives can amortize the cost of forward pass through the image encoder and provide denser gradients to the image encoder.
3. **[Related to above] multiple parallel decoders:**
The above suggestion can be extended to enable the use of multiple lightweight text decoders with multiple auxiliary objectives.
One can design each decoder head with reduced capacity to make all models have comparable sizes.
Training with such multiple objectives should speed up convergence.
4. **Evaluation on dense prediction tasks?**
If the goal of this study is to learn high-quality image encoders, then I suggest the authors may include additional evaluations
with dense prediction tasks like object detection and segmentation. These tasks are ubiquitous in vision and quite challenging.
I suggest that authors could train a ViTDet-style model with a frozen/fine-tunable image encoder from Cap/CapPa training.
Questions
Minor suggestion:
- This paper references CLIP in many places in the text. However, it is sometimes awkward to read as "[40] showed..." or "released by [40] ...".
Ultimately, it is personal preference, but I may recommend the authors use `citet` format like "Radford et al. [40] showed that ..."
for a better reading experience if they do not have a preference.
- `Line 84`: GeLU -> GELU. ReLU ("Re" = "Rectified") and GELU ("GE" -> "Gaussian Error") :-)
- What is the "(ok:as)" in section 4.2 title? Seems like a latex macro :-)
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
The authors have included a reasonable discussion about the limitations of this study (and their trained captioning models)
in the final section of the paper.