Memory Attentive Fusion: External Language Model Integration for Transformer-based Sequence-to-Sequence Model

This paper presents a novel fusion method for integrating an external\nlanguage model (LM) into the Transformer based sequence-to-sequence (seq2seq)\nmodel. While paired data are basically required to train the seq2seq model, the\nexternal LM can be trained with only unpaired data. Thus, it is important to\nleverage memorized knowledge in the external LM for building the seq2seq model,\nsince it is hard to prepare a large amount of paired data. However, the\nexisting fusion methods assume that the LM is integrated with recurrent neural\nnetwork-based seq2seq models instead of the Transformer. Therefore, this paper\nproposes a fusion method that can explicitly utilize network structures in the\nTransformer. The proposed method, called {\\bf memory attentive fusion},\nleverages the Transformer-style attention mechanism that repeats source-target\nattention in a multi-hop manner for reading the memorized knowledge in the LM.\nOur experiments on two text-style conversion tasks demonstrate that the\nproposed method performs better than conventional fusion methods.\n