Generative Pre-training for Paraphrase Generation by Representing and Predicting Spans in Exemplars

Paraphrase generation is a long-standing problem and serves an essential role\nin many natural language processing problems. Despite some encouraging results,\nrecent methods either confront the problem of favoring generic utterance or\nneed to retrain the model from scratch for each new dataset. This paper\npresents a novel approach to paraphrasing sentences, extended from the GPT-2\nmodel. We develop a template masking technique, named first-order masking, to\nmasked out irrelevant words in exemplars utilizing POS taggers. So that, the\nparaphrasing task is changed to predicting spans in masked templates. Our\nproposed approach outperforms competitive baselines, especially in the semantic\npreservation aspect. To prevent the model from being biased towards a given\ntemplate, we introduce a technique, referred to as second-order masking, which\nutilizes Bernoulli distribution to control the visibility of the\nfirst-order-masked template's tokens. Moreover, this technique allows the model\nto provide various paraphrased sentences in testing by adjusting the\nsecond-order-masking level. For scale-up objectives, we compare the performance\nof two alternatives template-selection methods, which shows that they were\nequivalent in preserving semantic information.\n