The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders

Multi-task learning with transformer encoders (MTL) has emerged as a powerful\ntechnique to improve performance on closely-related tasks for both accuracy and\nefficiency while a question still remains whether or not it would perform as\nwell on tasks that are distinct in nature. We first present MTL results on five\nNLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over\nsingle-task learning. We then conduct an extensive pruning analysis to show\nthat a certain set of attention heads get claimed by most tasks during MTL, who\ninterfere with one another to fine-tune those heads for their own objectives.\nBased on this finding, we propose the Stem Cell Hypothesis to reveal the\nexistence of attention heads naturally talented for many tasks that cannot be\njointly trained to create adequate embeddings for all of those tasks. Finally,\nwe design novel parameter-free probes to justify our hypothesis and demonstrate\nhow attention heads are transformed across the five tasks during MTL through\nlabel analysis.\n