Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation

Pre-trained language-vision models have shown remarkable performance on the\nvisual question answering (VQA) task. However, most pre-trained models are\ntrained by only considering monolingual learning, especially the resource-rich\nlanguage like English. Training such models for multilingual setups demand high\ncomputing resources and multilingual language-vision dataset which hinders\ntheir application in practice. To alleviate these challenges, we propose a\nknowledge distillation approach to extend an English language-vision model\n(teacher) into an equally effective multilingual and code-mixed model\n(student). Unlike the existing knowledge distillation methods, which only use\nthe output from the last layer of the teacher network for distillation, our\nstudent model learns and imitates the teacher from multiple intermediate layers\n(language and vision encoders) with appropriately designed distillation\nobjectives for incremental knowledge extraction. We also create the large-scale\nmultilingual and code-mixed VQA dataset in eleven different language setups\nconsidering the multiple Indian and European languages. Experimental results\nand in-depth analysis show the effectiveness of the proposed VQA model over the\npre-trained language-vision models on eleven diverse language setups.\n