CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

CLIP has shown a remarkable zero-shot capability on a wide range of vision\ntasks. Previously, CLIP is only regarded as a powerful visual encoder. However,\nafter being pre-trained by language supervision from a large amount of\nimage-caption pairs, CLIP itself should also have acquired some few-shot\nabilities for vision-language tasks. In this work, we empirically show that\nCLIP can be a strong vision-language few-shot learner by leveraging the power\nof language. We first evaluate CLIP's zero-shot performance on a typical visual\nquestion answering task and demonstrate a zero-shot cross-modality transfer\ncapability of CLIP on the visual entailment task. Then we propose a\nparameter-efficient fine-tuning strategy to boost the few-shot performance on\nthe vqa task. We achieve competitive zero/few-shot results on the visual\nquestion answering and visual entailment tasks without introducing any\nadditional pre-training procedure.\n