Long Document Summarization in a Low Resource Setting using Pretrained Language Models

Abstractive summarization is the task of compressing a long document into a\ncoherent short document while retaining salient information. Modern abstractive\nsummarization methods are based on deep neural networks which often require\nlarge training datasets. Since collecting summarization datasets is an\nexpensive and time-consuming task, practical industrial settings are usually\nlow-resource. In this paper, we study a challenging low-resource setting of\nsummarizing long legal briefs with an average source document length of 4268\nwords and only 120 available (document, summary) pairs. To account for data\nscarcity, we used a modern pretrained abstractive summarizer BART (Lewis et\nal., 2020), which only achieves 17.9 ROUGE-L as it struggles with long\ndocuments. We thus attempt to compress these long documents by identifying\nsalient sentences in the source which best ground the summary, using a novel\nalgorithm based on GPT-2 (Radford et al., 2019) language model perplexity\nscores, that operates within the low resource regime. On feeding the compressed\ndocuments to BART, we observe a 6.0 ROUGE-L improvement. Our method also beats\nseveral competitive salience detection baselines. Furthermore, the identified\nsalient sentences tend to agree with an independent human labeling by domain\nexperts.\n