Learning to Encode Position for Transformer with Continuous Dynamical Model

We introduce a new way of learning to encode position information for\nnon-recurrent models, such as Transformer models. Unlike RNN and LSTM, which\ncontain inductive bias by loading the input tokens sequentially, non-recurrent\nmodels are less sensitive to position. The main reason is that position\ninformation among input units is not inherently encoded, i.e., the models are\npermutation equivalent; this problem justifies why all of the existing models\nare accompanied by a sinusoidal encoding/embedding layer at the input. However,\nthis solution has clear limitations: the sinusoidal encoding is not flexible\nenough as it is manually designed and does not contain any learnable\nparameters, whereas the position embedding restricts the maximum length of\ninput sequences. It is thus desirable to design a new position layer that\ncontains learnable parameters to adjust to different datasets and different\narchitectures. At the same time, we would also like the encodings to\nextrapolate in accordance with the variable length of inputs. In our proposed\nsolution, we borrow from the recent Neural ODE approach, which may be viewed as\na versatile continuous version of a ResNet. This model is capable of modeling\nmany kinds of dynamical systems. We model the evolution of encoded results\nalong position index by such a dynamical system, thereby overcoming the above\nlimitations of existing methods. We evaluate our new position layers on a\nvariety of neural machine translation and language understanding tasks, the\nexperimental results show consistent improvements over the baselines.\n