Generate-Distill: Training Cross-Language IR Models with Synthetically-Generated Data

Abstract

Most pretrained language models that support neural information retrieval are fine-tuned on the MS MARCO dataset. MS MARCO is expressed in English, so it naturally supports monolingual English retrieval. However, for Cross-Language Information Retrieval (CLIR), no similar training data naturally exists that matches the languages of the query and the documents. The main ways to address this problem have been to continue to fine-tune with English data, or to translate MS MARCO queries and/or documents to match the CLIR setting. Machine translation often introduces errors that reduce retrieval effectiveness. It is usually easy to find target language documents suitable for training, but difficult to find naturally-occurring queries in the query language. An alternative is to train on naturally-occurring documents and synthetically-generated queries. Generate-Distill uses this approach with state-of-the-art distillation methods to match the effectiveness of training with translated MS~MARCO across different domains.