ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization

Abstractive text summarization is one of the areas influenced by the\nemergence of pre-trained language models. Current pre-training works in\nabstractive summarization give more points to the summaries with more words in\ncommon with the main text and pay less attention to the semantic similarity\nbetween generated sentences and the original document. We propose ARMAN, a\nTransformer-based encoder-decoder model pre-trained with three novel objectives\nto address this issue. In ARMAN, salient sentences from a document are selected\naccording to a modified semantic score to be masked and form a pseudo summary.\nTo summarize more accurately and similar to human writing patterns, we applied\nmodified sentence reordering. We evaluated our proposed models on six\ndownstream Persian summarization tasks. Experimental results show that our\nproposed model achieves state-of-the-art performance on all six summarization\ntasks measured by ROUGE and BERTScore. Our models also outperform prior works\nin textual entailment, question paraphrasing, and multiple choice question\nanswering. Finally, we established a human evaluation and show that using the\nsemantic score significantly improves summarization results.\n