Weaknesses
1. The results do not suggest that “ICL by AR pretraining is more difficult than ICL by few-shot pretraining”, as the authors claim. Instead, they suggest that ICL to predict the $T+1$-th token given the sequence $\mathbf{W}^0\mathbf{x}\_1, \dots, \mathbf{W}^{T-1}\mathbf{x}\_1$ is harder than ICL to predict $\mathbf{w}^\top \mathbf{x}_{T+1}$ given $\{(\mathbf{x}\_i, \mathbf{w}^\top \mathbf{x}\_i)\}\_{I=1}^T$ and $\mathbf{x}\_{T+1}$. These are two very different types of tasks and it is not fair to make conclusions about pretraining strategies based on their differing performance on these different tasks. Rather, one needs to compare the performance of a transformer trained in an autoregressive fashion and one trained on a few-shot learning objective under the same data/task model. In the data model in this, few-shot pertaining would use the same loss as the AR loss (equation 2) except that only the $t=T-1$ terms would remain in the loss. It is not clear whether this objective leads to better or worse solutions, in fact, this objective contains the same amount of information and therefore intuition suggests that it leads to solutions of the same quality. This is a key point as arguably the main message of the paper is that AR pretraining behaves differently than few-shot pertaining and thus the ICL community should focus on AR pretraining. I would suggest that the authors re-frame their discussion around the inadequacy of the regression data model considered by most of the existing ICL literature, as changing the data model is really the key contribution of this paper.
2. The setting is very artificial such that the significance of the results seems dubious. The model consists only of one, linear attention unit, and the data is assumed to follow a deterministic linear system, i.e. $\mathbf{x}\_{i+1}=\mathbf{Wx}\_i$ where $\mathbf{W}$ is diagonal with diagonal entries having magnitude 1. The learning model and data model both being linear means that the ICL prediction of any learning model with parameters $\mathbf{W}^{KQ}\_{32}=a\mathbf{I}_d$ and $\mathbf{W}^V\_{12} = b\mathbf{I}\_d$ for $ab>0$ and all other parameters being zero can be interpreted as the prediction of a linear model $\hat{\mathbf{W}}$ trained directly to approximate $\mathbf{W}$ with one step of GD starting from $\mathbf{0}\_{d\times d}$. It is not clear at all how such a conclusion would follow if the softmax activation, shown to be crucial in practice, is reinstated, or if other components of the transformer (MLP layer, multiple attention heads, multiple attention layers, etc.) are added. Also, the token embedding is strange and redundant, with consecutive inputs $\mathbf{x}\_i, \mathbf{x}\_{i-1}$ concatenated in the same token. Another, though less significant, issue is that only the population loss is considered. The authors repeatedly claim that “We believe that our analysis can give insight to the establishment of theory in settings with more complex models” but never explain why.
3. The model is already initialized as a model that implements gradient descent during ICL, and under the scalar-times-identity covariance assumption (4.1), or the enforced off-diagonal masking, it is not hard to show that the gradient flow dynamics reduce to the dynamics of two scalars ($a$ and $b$) which parameterize a set of models that always does GD during ICL. It is not clear whether the trained transformer can learn to perform GD during ICL if it is not initialized as a model that does this, and the experiments do not address this question.
4. “Gradient clipping” is used to refer to the masking procedure that zeros out all off-diagonal elements of the gradient. This undersells the severity of this procedure — clipping is a non-zero-thresholding operation applied equally to all gradient elements and is commonly used in practice for optimization and privacy purposes, and does not use any prior knowledge of how the gradient should be structured. The masking procedure described here sends a particular subset of elements of the gradient to zero, to drive the dynamics towards a particular desired solution based on prior knowledge of the data-generating process, which is a much stronger, and impractical, procedure.
5. Assumption 4.2 and Theorem 4.2 are tautological and not insightful. They should identify the properties of $\mathbf{x}\_1$ and $\mathbf{W}$ that lead to such behavior.
6. $\kappa\_3$ has $\Omega(d)$ growth, which means the effective step size of the gradient descent step is $\tilde{O}(T/d)$, meaning $T$ (number of examples in the context length) must be $\tilde{\Omega}(d)$ for accurate predictions, which makes sense for this problem setting, but suggests the proposed setting is far from capturing ICL ability in practice, where the number of in-context examples is far fewer than the token embedding dimension. In the experiments, $T=20d$, which means the experiments do not evaluate whether the results can extend to smaller-context scenarios.
7. Proposition 4.2 is not meaningful because it does not rule out whether a different initialization could lead to a model that performs GD.
8. Even if we accept all of the simplifications and assumptions (including very long context) made to obtain a trained model that performs GD during ICL, this trained model does not learn the correct step size for GD, so it can’t perform ICL. To me, this suggests that GD is *not* the mechanism that transformers use to perform ICL — even in such an idealized setting, GD cannot explain the ability of transformers to perform ICL. However, the messaging around the paper is not consistent with this, as it continues to push the narrative that trained transformers are mesa-optimizers.