ENCONTER: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer

Pretrained using large amount of data, autoregressive language models are\nable to generate high quality sequences. However, these models do not perform\nwell under hard lexical constraints as they lack fine control of content\ngeneration process. Progressive insertion-based transformers can overcome the\nabove limitation and efficiently generate a sequence in parallel given some\ninput tokens as constraint. These transformers however may fail to support hard\nlexical constraints as their generation process is more likely to terminate\nprematurely. The paper analyses such early termination problems and proposes\nthe Entity-constrained insertion transformer (ENCONTER), a new insertion\ntransformer that addresses the above pitfall without compromising much\ngeneration efficiency. We introduce a new training strategy that considers\npredefined hard lexical constraints (e.g., entities to be included in the\ngenerated sequence). Our experiments show that ENCONTER outperforms other\nbaseline models in several performance metrics rendering it more suitable in\npractical applications. Our code is available at\nhttps://github.com/LARC-CMU-SMU/Enconter\n