In this paper, we aim to build the global convergence theory of encoder-only shallow Transformers under a realistic setting from the perspective of architectures, initialization, and scaling under a finite width regime. The difficulty lies in how to tackle the softmax in self-attention mechanism, the core ingredient of Transformer. In particular, we diagnose the scaling scheme, carefully tackle the input/output of softmax, and prove that quadratic overparameterization is sufficient for global convergence of our shallow Transformers under commonly-used He/LeCun initialization in practice. Besides, neural tangent kernel (NTK) based analysis is also given, which facilitates a comprehensive comparison. Our theory demonstrates the separation on the importance of different scaling schemes and initialization. We believe our results can pave the way for a better understanding of modern Transformers, particularly on training dynamics.
Paper
Similar papers
Peer review
Summary
This paper proves the convergence rate of GD on one-layer transformer training. In their setting, the transformer is composed by a single attention layer, followed by relu unit and a fixed fully connected layer. The linear convergence rate is achieved. The proof technique follows the standard neural network proof [Nguyen 2021], where under some conditions of minimum singualr value of gram matrix, they inductively show that the parameter drift is controlled, and hence the GD dynamic is easy to study. The main meat is to show that the minimum singular value of gram matrix is lower bounded. [Nguyen 2021] Nguyen, Quynh. "On the proof of global convergence of gradient descent for deep relu networks with linear widths." In International Conference on Machine Learning, pp. 8056-8062. PMLR, 2021.
Strengths
To my best knowledge, this is the first convergence proof of transformer. The rate makes sense and proof roadmap is clear to me. The first part of the proof idea is standard, that under proper conditions for initialization and over-parameterization, the model parameter will stay in linear regime, and hence the convergence is easy to control. Then the second part is that they show that the conditions can be satisfied for some initialization scheme, by showing that the minimum singular value of gram matrix is bounded. The most interesting and novel part is to derive such lower bound.
Weaknesses
1. As I mentioned above, the first part of the proof (proposition 1) pretty much follows [Nguyen 2021]. 2. The model studied is bit toy. In practice, the residual block of transformer is necessary, but it is missing in this paper. 3. They assume the fully connected layer W_H is fixed to identical matrix, and argue that since W_V and W_H are adjacent, this is equavilent to W_H is trainable, However, in GD dynamic, I do not think they are equivalent. For example, in deep linear neural network, training all layer by GD is not equavilent to only training one matrix. Or if they are equivalent, there should be rigorous proof.
Questions
1. Since original framework in [Nguyen 2021] works when all layers are jointly trained, is it possible to extend your proof to that regime as well? 2. In Corollary 1, you said that when d_s and d is chosen to be some value, the conditions 2 and 3 cannot satisfied, and the convergence will fail. I think this is not a good reasoning. It could be that your framework cannot prove the convergence, but does not mean it cannot actually converge. A more convincing reasoning will be showing the lower bound of the loss, to show the failure of convergence.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The technical limitations are stated in weakness part. I do not see any negative social impact of this work.
Summary
This paper studies transformer networks consisting of one layer, with average pooling and a scalar output. The authors consider encoder type softmax attention (unmasked) and show convergence to a global minimizer of the loss. The results hold for the cases that the temperature/scaling inside the softmax $\tau_0$ is either inversely proportional to the overparameterization of the weight matrices, or the square root of it. The results also hold for different types of initialization He/Le-Cun. Finally, the authors also provide an NTK based analysis.
Strengths
To the best of my knowledge this is the first theoretical work on the convergence of transformers and could potentially lead to further investigate the convergence properties of transformers and bring insights into the way they work. It is also interesting that the authors study different initialization schemes and provide comparisons between them.
Weaknesses
Except for some questions I have (see below), my main concern is the formulation of the transformer model in this analysis. Specifically, 1. The paper does not study sequence to sequence models as encoder transformers normally are. Thus, I find the formulation of outputting a scalar inconsistent with practice. 2. The ReLU layers do not have any matrix after the application of the ReLU. There are also no residual connections. 3. The pooling layer to the best of my knowledge is used only in ViT models. I also think that it would be beneficial for the paper, to include more details about proof techniques for one of the cases (this could be possible by reducing some of the text in the rest of the sections).
Questions
1. In lines 635,637, do the authors mean the triangle inequality? 2. Why the authors study these two specific scaling schemes of $d_m^{-1/2}$ and $d_m^{-1}$? It would be interesting to have an analysis dependent on the scaling parameter $\tau_0$. 3. It is not very clear also what step-sizes could be permitted or constants $\hat{c}$. To my understanding the step-size needs to be inversely proportional to the number of data samples? It would be great if the authors could include some examples, in which the probability bounds are not vacuous and the order of the step-size chosen. 4. It would be great if the authors could include also the order of the constants $C_Q,C_K,C_V,C_O$ (if they cannot be chosen arbitrarily) or if they could point out the line in which they are defined. 5. In line 640, how is it implied that $\sigma_{min}(F_{pre}) \geq \alpha/2$?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
This work has no potential negative societal impact.
Summary
This paper presents global convergence results of shallow transformers (one self-attention layer followed by an MLP layer). The authors consider two different scalings for the factor $\tau_0$ used in the attention matrix as a function of $d_m$, the number of rows of $W_Q$ and $W_K$. Specifically, they prove that under LeCun/He/NTK initializations for the parameters of the Transformer, scaling $\tau_0$ as $d_m^{-1/2}$ requires quadratic over parametrization ($d_m = \Omega(N^2) $) to achieve convergence, while the $d_m^{-1}$ requires only linear over parametrization ($d_m = \Omega(N) $), though the later has a worse convergence rate. The paper claims that it is due the the lack of ability to capture pairwise interaction in the $\tau_0 = d_m^{-1}$ scaling. Numerical experiments are presented on synthetic data to corroborate their theoretical findings for the $d_m^{-1/2}$ scaling, as well as on MNIST, where the impact of choosing $\tau_0 = d_m^{-1/2}$ or $\tau_0 = d_m^{-1}$ on the training dynamics is discussed.
Strengths
1. This papers tackles the significant question of global convergence of Transformers. 2. The model considered is a realistic one, where both the attention module (involving pair-wise interaction between the $X^{i}$), and the MLP (applied independently on each Self-attention$(X)^{i}$ are considered (Eqs (1.1) and (1.2)). 3. The assumptions on the data-distribution are clearly discussed and Assumption 3. is verified in practice. Likewise, the assumptions of the parameters of the Transformer are clearly discussed. 4. Theorems 1 et 2 are strong results proving the global convergence of gradient descent of the shallow Transformer model for the general regression task studied in the paper. 5. The paper is focussed on the choice of the scaling factor inside the attention module, which is a forward steps towards understanding its impact. The theoretical findings as well as the experiments of the paper underlines the advantage of using the $d_m^{-1/2}$ scaling. 6. Numerical experiments are conducted to support the theoretical claims.
Weaknesses
1. Presentation: a. In my opinion the biggest weakness of the paper is its lack of clarity. The paper is not really well written, hard to read, with typos and missing words (e.g. l. 37 -42, 129, 152, 222). The paper would benefit from being rewritten in a clearer and more pedagogical manner. 2. Theoretical results: a. I read in Table 3 that SoftMax + ReLU leads to linear over parametrization. From the paper, we understand that this requires the scaling $d_m^{-1}$. However, as the authors remark it, this implies that the SoftMax operator will behave as a pooling layer. The link between the linear over-parametrization and the attention module behaving as a pooling layer is not clear. b. Lines 68 to 71. It is not clear whether the mentioned ReLU acts on each element of the sequence separately or mixes them together. 3. Experiments: a. The experiment on the synthetic data in the main paper only focuses on the $d_m^{-1/2}$ scaling. I see that the authors provide the results for the $d_m^{-1}$ scaling in the appendix. A discussion on the impact of the chosen scaling on the performance of the model on the synthetic data experiment is missing in the main paper. b. The paper focuses on regression while the experiment on MNIST seems to be a classification task. What plays the role of $y_n$ in this experiment ?
Questions
1. The quartic dependency of $N$ in the dimension $d$ is not discussed in Theorem 2. Is this a realistic assumption ? 2. Could the authors provide insights on why the $d_m^{-1}$ scaling, which leads the Attention module to behave as a pooling layer, requires only linear over parametrization ? 3. What happens if the Attention module is explicitly replaced by a pooling layer, that is $A_1 = ( 1 / d_s ) \times 1_{d_s \times d_s}X W_V^T$? 4. Could the authors comment the results of the synthetic data experiment for the $d_m^{-1}$ scaling ? And compare with the $d_m^{-1/2}$ scaling ? 5. Could the authors provide additional details for the MNIST experiment ? Is it a classification task ? What is the test-accuracy of the model ? What is the training loss when setting the attention module to the identity ? 6. Is there a setup where we should chose the $d_m^{-1}$ scaling ?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
Yes (Appendix F).
Summary
The transformer network is a popular but complicated architecture, which are widely used in NLP and computer vision. The contribution of this paper is two-folded: 1. it aims to give convergence guarantee for a shallow transformer network (encoder part) under different weight initialisations. Also, the paper concludes that the convergence rate of the NTK initialisation is slower than the original (LeCun/He) initialisation, which is often used in realistic setting; 2. it shows that a quadratic over-parametrisation guarantees global convergence of shallow transformer in usual setting. Assumptions are verified by realistic examples. Convergence and training dynamics are also tracked in both synthetic and real-world data sets.
Strengths
This paper gives a standard approach to proof global convergence by bounding the smallest singular value of the weight matrix in the last linear layer in the transformer network from above. The proof is delivered in a clear way, where notations and motivations are clearly stated. The lines seem to be flawless as I cannot find any typos or mistakes. This paper clearly has its significance in showing global convergence of transformer network in the original (LeCun/He) initialisation for the first time. Also, the over-parametrisation guarantee for global convergence is original, as far as I know. It also serves as a framework for later analysis in transformer.
Weaknesses
The paper only considers the shallow (encoder part of the) transformer network and training in gradient descent, which is still far away from practical architecture. Also, it would be good if I could see the comparison between theoretical and empirical convergence in a plot. Sadly, the plots only support the theorems qualitatively but not quantitatively.
Questions
The quantity $\alpha$ in Proposition 1seems to be sub-optimal. Is it the reason why you do not plot the theoretical decay with the empirical one? Could a log-scale plot be convincing? Also, in the proof of Lemma 1, you say by LLN we have the second equality in line (67). But the entries in $\beta'_j$ is correlated to these in $\beta_i$. How could you replace each vector respectively with $\frac{1}{d_s}\bm{1}_{d_s}$ by taking the limit before the product?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Yes, the authors include a section (Section F in appendix) to address the limitation of the paper.
Summary
This paper proves the global convergence of the shallow transformer under a more realistic setting. While I don't often read theoretical machine learning papers, I found some of the analysis interesting. However, the paper does feel a bit incremental compared to previous work.
Strengths
I would like to commend the authors for their efforts in conducting a more realistic analysis. I also appreciate their inclusion of analysis, discussion, experiments on artificial data, and experiments on real data, which makes the paper more complete.
Weaknesses
The paper does feel a bit incremental at times. It would be helpful to see more discussion of how the analysis could be applied to practical experiments.
Questions
In what way does the proposed analysis different from prior analysis, aparting from a more realistic setting? To achieve the more realist setting, do the authors have to come up with a different analysis technique to do that? It is not clear to me how novel the analysis techniques are.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Most of the analysis is done on shallow Transformers as pointed out by the authors. Moreover, the practical implication of the analysis is not yet clear.
Summary
The paper presents convergence results for shallow transformer networks for commonly used Gaussian-based initialization schemes in deep learning (LeCun, He) and different scaling regimes under finite overparameterization. In addition, the authors provide limiting neural tangent kernel (NTK) analysis for its minimum Eigenvalue and stability during training. They consider the encoder part of the transformer for the analysis which consists of a single-self attention layer, followed by ReLU, average pooling and a linear layer which maps to a scalar. The analysis is presented for gradient descent (GD) training with square loss applicable to classification/regression, where all parameters apart from the ReLU layer are trainable. The convergence framework for $1\sqrt{d_m}$ scaling uses three assumptions: bounded data, linearly independent tokens in each input example and dissimilarity of two different data examples. The $1/d_m$ scaling regime additionally uses a PDness assumption on the covariance matrix. The paper also provides experiments on synthetic gaussian data and on MNIST data for a Vision Transformer (ViT) tracking convergence under scaling regimes, width and NTK evolution.
Strengths
**Importance and motivation**: Very interesting work, I enjoyed reading it. The paper certainly takes a step and provides novel results towards understanding the training dynamics of shallow transformers and the overparameterization required for global convergence - an important research direction considering their prevalent usage. The authors do a good job in motivating the need to study different scaling regimes in the introduction section as well. **Organization and writing**: The paper is organized, well-written and easy to follow in general. **Assumption verification**: I like the important remarks and the experimental evidence authors use to back the assumptions, particularly for assumption 3 where they provide experiments on IMDB and MNIST dataset. Convergence results discussion in sec 4.5 on slower convergence of $1/d_m$ scaling and NTK initialization being slower than He/LeCun provide good insights into the theoretical results.
Weaknesses
**Missing intuitive explanations**: Standard convex analysis requires PL inequality + smoothness of the loss function to get a linear rate. Per my knowledge, the proof follows a similar(ish) recipe to [1], where they provide linear rates for NNs using the local gradient-Lipschitz property of the loss + a variant of PL, which gives a good analogue to convex analysis (as briefly mentioned already in the related work sec). I think it’d be nice for the reader if the paper contains a longer proof sketch with such intuitive comments/remarks for the proof steps. Some more suggestions on this can be found in the questions section. **Minor bugs and typos**: There are some minor issues in the proofs and some typos in the paper: please refer to questions. **Regarding assumption 3**: In its current form, assumption 3 forms a tail bound on the similarity of the respective empirical covariance matrices. This covariance inner product equals similarity of $\mathbf{X}_i, \mathbf{X}_j$ + some cross terms - how well does assumption 3 reflect dissimilarity as mentioned in the paper? It would have been nice had there been plots for similarity of $\mathbf{X}_i, \mathbf{X}_j$ along with the current ones. **Concerns/questions regarding the results**: I have several queries regarding separability constant $\hat{c}$, lower bound on $N$, etc. Please see the questions section. **Applicability on contextual token data**: It might be interesting to see if/how the results apply to the commonly used contextual token data setting [2, 3] for theoretical analysis of transformers. Eg: Assm. 2 fails if sparsity [3] isn’t small enough, a small discussion might be something to consider.
Questions
**Some more suggestions on intuitive remarks for proof sketch**: 1. It'd be helpful to mention the condition on gradient that makes the variant of PL and is used to prove convergence. 2. Breaking down (briefly) how the local Lipschitz property of loss is proved. **Typos/minor bugs**: 1. $ \gamma <=1/C$ in prop. Suggestion: It’d be helpful to specify that $C$ signifies local Lipschitz constant for the gradient. 2. Around line 653 in the appendix: Should it be $\theta^{t+1}, \theta^{t}$ instead of $\mathbf{W}^{t+1}, \mathbf{W}^{t}$? Also, in the same proof, second last step should be $\sigma_{min}^2$ and $||f^t-\mathbf{y}||^2$ - there might be a 2 factor missing in the result as well. 3. Lemma 15 proof, line 661: I don't think $\mathbf{XX}^T$ (correctly mentioned in Assm. 3 - $\mathbf{X}^T\mathbf{X}$) should be called the covariance matrix as the data is placed along rows. **Results**: 1. For thm. 1 remark: For larger $\hat{c}$, the exp quantity grows - which seems a bit counterintuitive in terms of separability of data. 2. In the $1/d_m$ regime there’s an additional req. On $N$, but not in the $1\sqrt{d_m}$ regime - is there any obvious reasoning for this or is it just a proof artifact 3. Is there any step-size $\gamma$ (or $C$, same thing) difference in $1/d_m$ vs $1/\sqrt{d_m}$? As far as I know, typically in NNs in the latter it is $O(1)$ [5] and $O(d_m)$ in the former [4]. 4. It’d be nice to know the authors' thoughts about the tightness on the width condition, especially in the $\sqrt{1/d_m}$ regime. Limitations section says the result doesn’t cover cross-entropy loss, can such a PL variant + local Lipschitz technique be applied to it? I’ve mostly seen such analysis with square loss [1, 5]. **References** [1]. Nguyen, Q.N. and Mondelli, M., 2020. Global convergence of deep networks with one wide layer followed by pyramidal topology. Advances in Neural Information Processing Systems, 33, pp.11961-11972. [2]. Li, H., Wang, M., Liu, S. and Chen, P.Y., 2023. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. arXiv preprint arXiv:2302.06015. [3]. Oymak, S., Rawat, A.S., Soltanolkotabi, M. and Thrampoulidis, C., 2023. On the Role of Attention in Prompt-tuning. arXiv preprint arXiv:2306.03435. [4]. Mousavi-Hosseini, A., Park, S., Girotti, M., Mitliagkas, I. and Erdogdu, M.A., 2022. Neural networks efficiently learn low-dimensional representations with sgd. arXiv preprint arXiv:2209.14863. [5]. Du, S.S., Zhai, X., Poczos, B. and Singh, A., 2018. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
I couldn’t find the code in the supplementary or any anonymous link to reproduce the results.
Thanks for the rebuttal
I would like to thank the authors for the clarification. The results in the paper is timely but given the dependence on previous work, I decide to maintain my score.
Response to authors
I would like to thank the authors for their response. After taking the time rereading the paper, I think that it is important to be rewritten such as to be more friendly to the reader. Specifically, to my opinion 1. Previous results should be at least cited if not restated so that the reader could easily find them. 2. The appendix should be self-contained. For example, terms like $f_{pre}$ and $F_{pre}$ should be restated and not used from the main text. 3. The lemmas in the appendix are entangled with each other, creating a sense of confusion. The statements should be more clear and concise. These are some comments that I hope to help the authors improve the writing of the paper. My main concern remains the formulation of the transformer architecture as well as the technical contribution of the paper raised by reviewer UvAq. However, I am raising my score to 6 given that the authors will create a revised version.
Response to reviewer sZxG
We are thankful to the reviewer sZxG for the prompt response and positive support. According to your suggestions, we will re-organize related work and appendix for better understanding. To be specific, - [Related work] The subsection ‘over-parameterization for convergence analysis’ in the related work will be elaborated with more details to describe Table 3 of the appendix. - [Self-contained appendix] We will include a more detailed notation in Table 2. We will also restate the meaning of each symbol if it was only mentioned in the main body. - [Entanglement of the lemmas in appendix]. We will add more details about theorems and lemmas, and a proof flowchart to point out the relationship between each lemma. Besides, the technical difficulty and proof sketch will be added to the main text for better understanding, to make our contributions more clear. We appreciate the feedback from the reviewer and we are open to more suggestions to improve the readability and contribution of our work. Best regards, Authors
Thanks a lot for your rebuttal and clarifications. After re-reading the paper in details, I still believe that there is a lack of clarity in both the formulation and the way the results are presented. These are not just minor typos. **Formulation** Some sentences are not grammatically correct: e.g.: l. 228: it is common to make either the assumption that the covariance of x is an identity matrix or weak assumption, e.g., positive definite. l. 245: Besides, the stability of NTK during training, which allows us to build the connection to kernel regression predictor. There are other examples. It makes things harder to understand for the reader. **Presentation of the results** I still believe the paper lacks pedagogy in presenting the results. For instance: 1. Proposition 1 has many notations and deserves some comments for the reader. 2. In 4.4 you propose an NTK analysis. But in 4.2 and 4.3 you also consider the NTK initialization under the two scalings for $\tau_0$, which is confusing. 3. You don't specify which scaling you use in Th.3 and 4. 4. The learning algorithm is never properly introduced. Therefore, my opinion is that this paper deserves to be rewritten, where formulation has been revised everywhere and results are presented in a clearer way. I strongly believe that this would be beneficial for the paper which is a good theoretical contribution.
improve the presentation according to Reviewer p3n5's writing suggestions
We thank the reviewer p3n5 for the constructive suggestions on writing. We have already corrected the grammatical errors, and imprecise expressions, in order to improve the readability. Indicatively, we list the revisions to the reviewer’s comments below: --- - l. 228: it is common to make either the assumption that the covariance of x is an identity matrix or weak assumption, e.g., positive definite. -> Frequently, the covariance of x is assumed to be an identity matrix or positive definite. - l. 245: Besides, the stability of NTK during training, which allows us to build the connection to kernel regression predictor. -> Besides, the stability of NTK during training allows us to build a connection on training dynamics between the Transformer (assuming a squared loss) and the kernel regression predictor. --- Regarding the presentation, according to your suggestions, we will introduce more descriptions when presenting our theoretical results. To be specific: --- **Description of Proposition 1:** We will modify lines 170-171 for clarity as follows: Define the following quantities at initialization for simplification: - The norm of the parameters: $\bar{\lambda}_Q = XXX$, $\bar{\lambda}_K= XXX$, $\bar{\lambda}_V= XXX$, $\bar{\lambda}_O= XXX$ - Two auxiliary terms: $\rho = XXX$ and $z = XXX$. Under Asm. 1, we assume that the minimum singular value of ${\bf F}\_{\text{pre}}^0$, i.e., $\alpha \triangleq \sigma\_{min}({\bf F}\_{\text{pre}}^0$) satisfies the following condition at initialization --- These variables will be included in Table 2 for better reference. Besides, more remarks will be added for a better understanding, e.g., how $\alpha$ is affected by different initialization schemes and the selection of the step-size. --- **Confusion on “NTK initialization/scaling” name:** The title of Sec. 4.4 will be modified as follows: 'NTK Analysis of Transformers' -> NTK Analysis with $\tau_0 = d_m^{-1} $' for better understanding. --- **Scaling used in Thm 3. and 4:** In these two theorems (as well as Section 4.1), the $\tau_0 = d_m^{-1}$ setting is employed. We will make it clear. --- **No description on learning algorithms:** We will include the following standard gradient descent algorithm for Transformer training in the revised version. It involves gradient updates for Transformer parameters ${\bf W}_Q$, ${\bf W}_K$, ${\bf W}_V$, ${\bf W}_O$ at the $t$-step: (denote ${\bf \theta}^0 := \\{ {\bf W}\_Q^0, {\bf W}\_K^0, {\bf W}\_V^0, {\bf W}\_O^0 \\}$ for simplicity): - ${\bf W}\_Q^{t+1} = {\bf W}\_Q^{t}- \gamma\cdot \nabla\_{{\bf W}\_Q} \ell ({\bf \theta}^t)$ - ${\bf W}\_K^{t+1} = {\bf W}\_K^{t}- \gamma\cdot \nabla\_{{\bf W}\_K} \ell ({\bf \theta}^t)$ - ${\bf W}\_V^{t+1} = {\bf W}\_V^{t}- \gamma\cdot \nabla\_{{\bf W}\_V} \ell ({\bf \theta}^t)$ - ${\bf W}\_O^{t+1} = {\bf W}\_O^{t}- \gamma\cdot \nabla\_{{\bf W}\_O} \ell ({\bf \theta}^t)$ --- Apart from the aforementioned revision, we will add the proof sketch, more details on theoretical results and experimental settings, , which has been mentioned in the previous response and [general response](https://openreview.net/forum?id=8ZveVHfmIE¬eId=hT7qrLWaaA). We will try our best to improve the readability of this paper and hope our clarifications address your concern. --- We sincerely appreciate the reviewer’s proofreading, which will definitely be beneficial to our paper.
Increasing my score to 5.
Dear authors, Thanks for the clarification. Given your commitment to significantly improve the writing and presentation of the results, I increase my score to 5.
Thank you to the authors for the detailed response to my comments, suggestions and the rebuttal. I am happy to keep my score of 7.
Decision
Accept (poster)