PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation

Despite recent progress of pre-trained language models on generating fluent\ntext, existing methods still suffer from incoherence problems in long-form text\ngeneration tasks that require proper content control and planning to form a\ncoherent high-level logical flow. In this work, we propose PLANET, a novel\ngeneration framework leveraging autoregressive self-attention mechanism to\nconduct content planning and surface realization dynamically. To guide the\ngeneration of output sentences, our framework enriches the Transformer decoder\nwith latent representations to maintain sentence-level semantic plans grounded\nby bag-of-words. Moreover, we introduce a new coherence-based contrastive\nlearning objective to further improve the coherence of output. Extensive\nexperiments are conducted on two challenging long-form text generation tasks\nincluding counterargument generation and opinion article generation. Both\nautomatic and human evaluations show that our method significantly outperforms\nstrong baselines and generates more coherent texts with richer contents.\n