Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation

Models pretrained with self-supervised objectives on large text corpora\nachieve state-of-the-art performance on English text summarization tasks.\nHowever, these models are typically fine-tuned on hundreds of thousands of data\npoints, an infeasible requirement when applying summarization to new, niche\ndomains. In this work, we introduce a novel and generalizable method, called\nWikiTransfer, for fine-tuning pretrained models for summarization in an\nunsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained\nmodels on pseudo-summaries, produced from generic Wikipedia data, which contain\ncharacteristics of the target dataset, such as the length and level of\nabstraction of the desired summaries. WikiTransfer models achieve\nstate-of-the-art, zero-shot abstractive summarization performance on the\nCNN-DailyMail dataset and demonstrate the effectiveness of our approach on\nthree additional diverse datasets. These models are more robust to noisy data\nand also achieve better or comparable few-shot performance using 10 and 100\ntraining examples when compared to few-shot transfer from other summarization\ndatasets. To further boost performance, we employ data augmentation via\nround-trip translation as well as introduce a regularization term for improved\nfew-shot transfer. To understand the role of dataset aspects in transfer\nperformance and the quality of the resulting output summaries, we further study\nthe effect of the components of our unsupervised fine-tuning data and analyze\nfew-shot performance using both automatic and human evaluation.\n