Summary
The paper identifies a simple task for which the Transformer architecture fails. The authors introduce flip-flop language modeling, towards quantifying the extrapolation capabilities of different architectures. Transformer models are determined to suffer from long tail errors, in a phenomenon termed "attention glitches".
Throughout the paper, this new synthetic task is analysed and the failure modes of Transformer models are identified. The authors embark and compare different techniques to mitigate the failures. Some theoretical justification and some mechanistic interpretability are also provided.
Strengths
1. The paper discusses a very simple task for which recurrent networks triumph, but Transformer present failure cases, attributing these failures to the inductive bias of the model. The task proposed is very simple and adequately motivated.
2. Long range dependencies are a common failure case of Transformer models. FFLM allows for easy to check failures for reasoning tasks.
3. Extensive experiments with different regularizations are provided.
Weaknesses
1. I feel some of the implications and connections to Ji et al., 2023 are not highlighted. From Figure 3, depth does not seem to have a direct consequence on performance.
2. Although some theoretical justification is provided, assumptions are in cases too strong, i.e. linear positional encoding.
3. Although a series of regularization and other techniques are proposed to solve the errors, no clear solution is proposed that works.
Questions
1. It will be helpful to report errors as a function of difficulty. This can be done by presenting the errors as a function of sequence length (as done in Figure 3-right) or by also presenting errors as a function of different FFL tasks. If we define as $d$, the distance between the last write and the current read operation, is there a connection between the distribution of $d$ in the training set and the types of errors in the extrapolation set? How many examples of distance $d$ need to be presented for the model to generalize? For (possible) outliers $d$ in the training set, is the model fitting them correctly? There might be some interesting connections with Grokking [1, 2], where the model switches from a state of memorizing to one of generalizing.
2. Preposition 2 states that a 2-layer 1-head Transformer can represent FFL. Is there is practice any benefit with depth?
3. Other ways to decrease the entropy of the attention include the change of the temperature inside the softmax (which is not included in Proposition 3), or the method of [3].
Fixes:
Figure 3 (left) x-axis is cropped (50 -> 500), also middle-right (10->10K).
[1] Power, Alethea, et al. "Grokking: Generalization beyond overfitting on small algorithmic datasets." arXiv preprint arXiv:2201.02177 (2022).
[2] Thilak, Vimal, et al. "The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon." arXiv preprint arXiv:2206.04817 (2022).
[3] Martins, André, et al. "Sparse and continuous attention mechanisms." Advances in Neural Information Processing Systems 33 (2020): 20989-21001.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper proposes a simple example to study failure in extrapolation of Transformer models. The authors identify the lack of correct inductive bias in the model as the major shortcoming. Although many different regularization techniques are proposed, no clear answer is given for this phenomenon. I believe that this will however steer the community towards the right direction, for answering fundamental questions what these models is concerned.