Response to Official Review of Submission7032 by Reviewer fT87
**[W1]**
We thank the reviewer for the suggestion. We would like to note that we do not claim that “contrastively pretrained visual encoder is superior to a classification-pretrained one for all purposes”. Instead, we reported our observations on contrastively trained SigLIP that “despite SigLIP slightly underperforming on image classification, it performs much better on Visual Language tasks.” (See our contribution #1) with a focus on the relative advantage of contrastively trained models on VL tasks, which we believe holds true for similarly trained models including OpenCLIP.
For OpenCLIP, We have not tried it at the time of submission, since SigLIP is also open-source and significantly better across benchmarks. However, we agree this can be an interesting ablation, for which we have started to set up and are trying to add an additional baseline based on OpenCLIP before the final version of the draft.
**[W2]**
We expect the zero-shot performance to be relatively poor, since there is no task aimed for this in the pre-training mixture, and hence the model cannot know "the coco vocabulary" from which it should generate. PaLI-3 is also not a chatbot, but rather a (compact) foundation model capable to be transferred to various tasks.
Meanwhile, few-shot in-context ability is a good idea that we could and likely should add in the pre-training mixture, and would provide an in-between alternative between fine-tuning and zero-shot.
**[W3]**
Thanks for the question, it highlights an unclarity in the write-up. The stage0 pre-training does not use a UL2 encoder-decoder. It is trained with a transformer encoder for the text with a similar size to ViT-L, following SigLIP (and like CLIP), which we then discard in the following stages. Thus, no such consistency in the language model is required. We will emphasize this aspect better in the paper’s next version.