VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Recently, fine-tuning language models pre-trained on large text corpora have\nprovided huge improvements on vision-and-language (V&L) tasks as well as on\npure language tasks. However, fine-tuning the entire parameter set of\npre-trained models becomes impractical since the model size is growing rapidly.\nHence, in this paper, we introduce adapter-based parameter-efficient transfer\nlearning techniques to V&L models such as VL-BART and VLT5. We evaluate our\nmethods in a unified multi-task setup on both image-text and video-text\nbenchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2,\nGQA, NLVR2 , and MSCOCO image captioning. For video-text tasks, we use TVQA,\nHow2QA, TVC, and YC2C. With careful training and thorough experiments, we\nbenchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter)\nagainst the standard full fine-tuning and the recently proposed prompt-tuning\napproach. We also enhance the efficiency and performance of adapters by sharing\ntheir weights to attain knowledge across tasks. Our results demonstrate that\ntraining the adapter with the weight-sharing technique (4.18% of total\nparameters for image-text tasks and 3.39% for video-text tasks) can match the\nperformance of fine-tuning the entire model. Lastly, we present a comprehensive\nanalysis including the combination of adapter and task-specific prompts and the\nimpact of V&L pre-training on adapters. Our code is available at:\nhttps://github.com/ylsung/VL_adapter.\n