Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence

This paper proposes a transformer over transformer framework, called\nTransformer$^2$, to perform neural text segmentation. It consists of two\ncomponents: bottom-level sentence encoders using pre-trained transformers, and\nan upper-level transformer-based segmentation model based on the sentence\nembeddings. The bottom-level component transfers the pre-trained knowledge\nlearned from large external corpora under both single and pair-wise supervised\nNLP tasks to model the sentence embeddings for the documents. Given the\nsentence embeddings, the upper-level transformer is trained to recover the\nsegmentation boundaries as well as the topic labels of each sentence. Equipped\nwith a multi-task loss and the pre-trained knowledge, Transformer$^2$ can\nbetter capture the semantic coherence within the same segments. Our experiments\nshow that (1) Transformer$^2$ manages to surpass state-of-the-art text\nsegmentation models in terms of a commonly-used semantic coherence measure; (2)\nin most cases, both single and pair-wise pre-trained knowledge contribute to\nthe model performance; (3) bottom-level sentence encoders pre-trained on\nspecific languages yield better performance than those pre-trained on specific\ndomains.\n