Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks

Loading models pre-trained on the large-scale corpus in the general domain\nand fine-tuning them on specific downstream tasks is gradually becoming a\nparadigm in Natural Language Processing. Previous investigations prove that\nintroducing a further pre-training phase between pre-training and fine-tuning\nphases to adapt the model on the domain-specific unlabeled data can bring\npositive effects. However, most of these further pre-training works just keep\nrunning the conventional pre-training task, e.g., masked language model, which\ncan be regarded as the domain adaptation to bridge the data distribution gap.\nAfter observing diverse downstream tasks, we suggest that different tasks may\nalso need a further pre-training phase with appropriate training tasks to\nbridge the task formulation gap. To investigate this, we carry out a study for\nimproving multiple task-oriented dialogue downstream tasks through designing\nvarious tasks at the further pre-training phase. The experiment shows that\ndifferent downstream tasks prefer different further pre-training tasks, which\nhave intrinsic correlation and most further pre-training tasks significantly\nimprove certain target tasks rather than all. Our investigation indicates that\nit is of great importance and effectiveness to design appropriate further\npre-training tasks modeling specific information that benefit downstream tasks.\nBesides, we present multiple constructive empirical conclusions for enhancing\ntask-oriented dialogues.\n