MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale

We study the zero-shot transfer capabilities of text matching models on a\nmassive scale, by self-supervised training on 140 source domains from community\nquestion answering forums in English. We investigate the model performances on\nnine benchmarks of answer selection and question similarity tasks, and show\nthat all 140 models transfer surprisingly well, where the large majority of\nmodels substantially outperforms common IR baselines. We also demonstrate that\nconsidering a broad selection of source domains is crucial for obtaining the\nbest zero-shot transfer performances, which contrasts the standard procedure\nthat merely relies on the largest and most similar domains. In addition, we\nextensively study how to best combine multiple source domains. We propose to\nincorporate self-supervised with supervised multi-task learning on all\navailable source domains. Our best zero-shot transfer model considerably\noutperforms in-domain BERT and the previous state of the art on six benchmarks.\nFine-tuning of our model with in-domain data results in additional large gains\nand achieves the new state of the art on all nine benchmarks.\n