Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals

Transformers have achieved remarkable success in a wide range of natural language processing and computer vision applications. However, the representation capacity of a deep transformer model is degraded due to the over-smoothing issue in which the token representations become identical when the model's depth grows. In this work, we show that self-attention layers in transformers minimize a functional which promotes smoothness, thereby causing token uniformity. We then propose a novel regularizer that penalizes the norm of the difference between the smooth output tokens from self-attention and the input tokens to preserve the fidelity of the tokens. Minimizing the resulting regularized energy functional, we derive the Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO), a novel class of transformer models that can mitigate the over-smoothing issue. We empirically demonstrate the advantages of NeuTRENO over the baseline transformers and state-of-the-art methods in reducing the over-smoothing of token representations on various practical tasks, including object classification, image segmentation, and language modeling.

Paper

References (71)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer k2Mg7/10 · confidence 3/52023-06-17

Summary

Recent research works have revealed that the over-smoothing issue, a prevalent challenge in Graph Neural Networks, similarly plagues Transformers. Contrary to expectations, the performance of a Transformer model does not invariably improve with increased depth of the self-attention layers. In fact, a deeply layered Transformer may not necessarily outperform a shallow one. This paper elucidates that the over-smoothing phenomenon occurs due to a consistent decrease in the non-local function of weighted differences in each self-attention layer as depth increases. Drawing on this finding, the paper puts forward a solution featuring modified skip connections. While similar solutions have been proposed in prior research, this paper sets itself apart by providing an in-depth theoretical analysis from a gradient flow perspective, which previous works have not addressed. Furthermore, the experimental results corroborate the effectiveness of the proposed solution in mitigating over-smoothing in Transformers.

Strengths

1. The theoretical investigation conducted in this paper is rigor and in-depth.

Weaknesses

1. The theoretical explication put forth in this paper applies accurately only to Transformers with single-head self-attention mechanisms. However, given that the prevailing Transformers utilize multi-head self-attention mechanisms and feed-forward networks, it becomes prudent to critically evaluate the efficacy of the resolution proposed herein. 2. This paper lacks a comprehensive set of comparative experiments to analyze the influence of diverse initial values of the parameter $\widetilde{\lambda}$ on the model's final performance.

Questions

1. This paper introduces a concept of symmetric self-attention, i.e., the query matrix is equal to the key matrix. The experimental results demonstrate an unexpectedly high performance from a Transformer predicated on symmetric self-attentions. This performance seems to contravene the conventional understanding that self-attention should capture the bi-directional correlation among tokens. Can the authors provide a logical elucidation to account for these particular experimental outcomes?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

N/A

Reviewer qcqa6/10 · confidence 3/52023-07-03

Summary

This work shows that self-attention layers in transformers minimize a funcitonal which promotes smoothneess, thereby casuing token uniformity. The work also proposes a novel regularizer to preserve fidelity of the tokens. The work empirically shows that NeuTRENO outperforms baseline transformers in reducing the over-smoothing of token representation.

Strengths

- I like the flow of the paper. The authors dive straight into the key issue of the work, without much retrospect on the previous known matters. - The paper rewrites the self-attention as a Gradient Descent Step to minimizie a nonlocal functional. - The paper shows that as k (number of layers) increases, the model is more likely to suffer from over-smoothing. - I think viewing each self-attention layer in the transformer as an impliciit gradient ascent in very interesting. It echos with some literature in the past, providing more insights on the interpretability of deep neural networks.

Weaknesses

- Some use of the wording could be improved, for example in line 220s, the authors like to use words like "significantly", "addresses" etc. In my opinion, some of the experimental results are not strong enough to support author's claim. It could create misleading for the audiences. - Experiments looks very interesting. I think the paper might benefit more by discussing broader impact of "alleviating over-smoothness". Say for example. would be help in OOD setting? incremental setting? or other interesting applications. In my opinion, 1% increase on standard metric should not be the best selling point for this interesting work.

Questions

- Although the authors motivates this work from the point of over-smoothing, the paper lacks discussion on the actual cause, impact and consequence of over-smoothing. For example, from Figure.1, we observe that NeuTRENO has a smaller cosine similarity comparing to naive DeiT. One question would be, what should be the desired cosine sim? Is smaller cosine sim siginificantly better than say sim over 0.8?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Please see the previous two sections

Reviewer h3ak6/10 · confidence 4/52023-07-06

Summary

This paper analyses the over-smoothing problem of transformer architecture by showing that self-attention layers minimize a functional that causes over-smooth. To address this problem, the authors introduce a regularizer that penalizes the norm of the difference between the smooth output tokens and input tokens. In the experimental section, the authors demonstrate that their proposed solution, NeuTRENO, outperforms baseline models on visual and language-related downstream tasks.

Strengths

The issue of over-smoothing in transformers is a fascinating topic. While previous research has explored this problem, this paper offers a unique perspective by examining self-attention as a gradient descent step from a variational standpoint. This novel approach sheds new light on the over-smoothing problem and contributes to a deeper understanding of the issue within the research community. The effectiveness of the proposed solution, NeoTRENO, in mitigating the problem of over-smoothing is demonstrated both theoretically and empirically. The experiments show NeoTRENO's effectiveness in both vision and language transformers.

Weaknesses

Current experiments are mostly conducted on small transformer models( e.g., DeiT-tiny). Given that the transformer architecture is crucial for large pre-trained language models, it remains unclear whether the proposed solution can effectively alleviate the over-smoothing problem when the model size is increased, and whether can be combined with large pre-trained language models (e.g., BERT).

Questions

please refer to weaknesses

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The generalizability and scalability of the proposed solution to larger transformer models have not been investigated.

Reviewer YNWk5/10 · confidence 4/52023-07-07

Summary

This paper studies the oversmoothing problem in transformers. Roughly speaking, it was observed that embeddings start to converge when the network is deeper. The authors built a model to explain this phenomenon. Roughly speaking, they relate having deeper architecture with making progress towards minimizing a function. Then they built some regularizers out of this intuition.

Strengths

I think these analysis and regularization tools are useful addition to the exploding area of transformers.

Weaknesses

I feel I was not able to properly parse some math texdt. My most unsure part is whether the amount of hand-waiving is too much even under today’s standard. For example, the authors feel very comfortable in exchanging integration operator and summation. It is true in general the approximation errors are usually inconsequential. But it is still different from pretending that they are the same in theorem statements/proofs. Also, I dont understand how K(x, y), k(x, y) and boldk(x) and boldk(y) are related. First k(x, y) are defined, then there will always exist boldk? It also looks that sometimes k/K are treated as kernels whereas sometimes it is treated as a kernel function. Probability functions and kernel functions have different properties?

Questions

It was proven that an update of a self-attention is equivalent to taking a gradient over a function. But will the function have many local optimal so that even we have the main theorem, it does not quite say all the embeddings will eventually be the same? Or maybe the function is a convex function.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Reviewer qvvE4/10 · confidence 5/52023-07-10

Summary

This paper analyzes the reason of over-smoothing in Transformer based on Self-attention as a Gradient Descent Step to Minimize a Nonlocal Function and random walk analysis. The paper then proposes a novel regularizer that penalizes the norm of the difference between the output tokens from self-attention and the input tokens to preserve the fidelity of the tokens. Experimental results on ImageNet classification, image segmentation and LM tasks show that the proposed approach achieves better performance than the baseline vanilla Transformer.

Strengths

Strengths: (1) The paper provides interesting aspects to analyze the reason of over-smoothing in Transformers based on Self-attention as a Gradient Descent Step to Minimize a Nonlocal Function and random walk analysis. (2) The theoretical proofs are detailed and clear.

Weaknesses

Weaknesses: (1) Although the paper provides detailed theoretical proofs of the nonlocal variational denoising framework for self-attention and provides an explanation for the over-smoothing issue in transformer-based models, the empirical evaluations missed some important details and analyses. In Table 1, the configuration of NeuTRENO Adaptation is not explained. (2) Except the results in Table 2 on image segmentation, the gains from the proposed approach in Table 1 ImageNet classification and WikiText-103 LM are all quite small. In Table 1, the gains from the proposed approach over the baseline are 0.84 on Top-1 acc and 0.54 on Top-5 acc. In Table 3, the gains from the proposed approach over baseline are 0.55 on validation PPL and 0.59 on test PPL. It is not clear how stable the proposed model is on these datasets, since there is no reporting of standard deviations from multiple runs with different random seeds, so it is not clear whether the gains are by chance. Also, it is not clear whether these gains are statistically significant. (3) There have been prior works analyzing the uniformity and alignment problem of BERT on sentence representations and proposed postprocessing solutions, for example, the flow https://arxiv.org/pdf/2011.05864.pdf and BERT whitening method https://arxiv.org/abs/2103.15316 and supervised method such as SimCSE https://arxiv.org/abs/2104.08821 and enhancement over SimCSE on sentence representations. These works are not compared to or cited in the paper. (4) An important point is missing. Based on the results from models addressing uniformity in BERT and improving sentence representations, these models alleviate uniformity but cause degradations in transfer learning capability, e.g., SimCSE. This issue has not been discussed in the paper since the paper did not investigate the proposed approach in pre-training and fine-tuning paradigm, to investigate the impact of the proposed approach on transferability. (5) There are some presentation problems. Some math symbols and functions are not defined or clearly defined, as listed under Questions.

Questions

Please address the points under weaknesses. Also, there are some presentation errors in the paper. For example, (1) The word “functional” is an adjective, so using it as a noun in the paper is ungrammatical, for example, in the title and phrases such as “minimize a functional which…” “Minimizing the resulting regularized energy functional, …” throughout the paper. (2) Some math symbols have not been defined, for example, D_x , D_{qk} in Section 1.1. (3) Line 89 defines the weights k(x,y), but the definition is kind of vague. Line 117 provides a form of K(x,y) :=k(x,y)+k(y,x), which needs to be explained clearer about this choice of K(x,y).

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

This is N/A for this work.

Reviewer qcqa2023-08-16

I thank the author for providing this rebuttal. I agree that it is a very interesting work that tried to understand the self-attention of transformers as a gradient descent step. I am still a bit confused by the over-smoothness sections. For example, in figure.7 of the rebuttal pdf, the proposed new architecture seem to successfully decrease the cosine sim between tokens. But does that lead to better empirical performance or better interpretability? On vision tasks, I am still dubious about the statement that over-smooth is indeed a issue and should be addressed. Of course, I think the work is very valuable in terms of trying to address this problem if this problem is indeed very significant.

Authorsrebuttal2023-08-17

Response to Reviewer qcqa: Oversmoothing and Empirical Performance

Thanks for your further feedback. Please allow us to address your concerns below. We will include the following discussion and existing works in our revised manuscript to clarify the impact of oversmoothing on the model’s performance, especially for the vision transformer (ViT). **Question: For example, in figure.7 of the rebuttal pdf, the proposed new architecture seems to successfully decrease the cosine sim between tokens. But does that lead to better empirical performance or better interpretability?** **Reply:** In Figure 7 (Left) of the rebuttal PDF, our NeuTRENO BERT finetuned on the SQUAD question answering task [1] yields better accuracy than the baseline BERT finetuned on the same task (81.39 exact match score and 88.62 F1 score vs. 80.77 exact match score and 88.12 F1 score). This result indicates that reducing cosine similarity between tokens in the trained transformer-based models leads to better empirical performance. **Question: On vision tasks, I am still dubious about the statement that over-smooth is indeed an issue and should be addressed.** **Reply:** The oversmoothing in ViT has been verified and investigated in existing works. In particular, [2] observes that the performance of ViT quickly saturates as more layers are added to the model. Moreover, experiments in [2] show that the 32-layer ViT underperforms the 24-layer ViT, indicating the difficulty of ViTs in gaining benefits from deeper architectures. The authors point out that oversmoothing results in this phenomenon by causing the token representations to become identical when the model grows deeper. Based on this observation, they propose a cross-head communication method that helps enhance the diversity of both token representations and attention matrices. Furthermore, it has been shown in [3] that the training of ViT models encounters instability with greater depths. [4] proposes that this instability arises from the oversmoothing, where token representation for patches within an image become progressively alike as the model's depth increases. In an effort to explain this issue, [5] finds out that self-attention acts as a low-pass filter, which smoothens the token representations in ViTs. This leads to the proposal of the FeatScale method [5], which regulates feature frequencies, whether low or high, to counteract the consequences of oversmoothing. Different from existing works, in our paper, we theoretically prove the oversmoothing phenomenon via the variational denoising framework that we develop. Our proposed NeuTRENO method helps mitigate the oversmoothing issue and improves the performance of the ViT baselines (DeiTs), as well as other baseline transformer-based models. We also empirically demonstrate that NeuTRENO not only reduces the cosine similarity between token representation (See Figure 1 in the main text and Figure 7) but also the redundancy in attention maps between layers (See Figure 5 in Appendix G2). Finally, we show that NeuTRENO is complementary to existing methods, including FeatScale [5] (See Table 1 and Table 7 in our paper). **References** [1] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. “SQuAD: 100,000+ questions for machine comprehension of text”. EMNLP, 2016. [2] Daquan Zhou, et al. “Deepvit: Towards deeper vision transformer”. arXiv, 2021. [3] Touvron Hugo, et al. "Going deeper with image transformers." ICCV, 2021. [4] Chengyue Gong, et al. “Vision transformers with patch diversification”. arXiv, 2021. [5] Wang Peihao, et al. "Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice." ICLR, 2022.

Authorsrebuttal2023-08-19

Response to Reviewer qcqa: Oversmoothing and Empirical Performance (Continued with Additional Results)

Following your suggestion, we have obtained additional results to further address your concern about the impact of oversmoothing on the model's performance. In particular, in Figure 7 (Right) in the rebuttal PDF, our NeuTRENO SimCSE, after fine-tuned on the STS-12 semantic textual similarity task [1], gains a significant improvement over the baseline SimCSE [2], which is also fine-tuned on the same task (77.32% vs. 75.29% Spearman's correlation. Here, the higher correlation, the better). This additional result further verifies that decreasing the cosine dissimilarity between tokens within trained transformer-based models leads to improved empirical performance. **References** [1] Eneko Agirre, et al. "SemEval-2012 task 6: A pilot on semantic textual similarity." SemEval, 2012. [2] Gao, T., Yao, X., and Chen, D. "SimCSE: Simple Contrastive Learning of Sentence Embeddings." EMNLP, 2021.

Reviewer qcqa2023-08-20

Thank the authors for providing these additional clarifications. Overall the contribution in this paper is non-trivial and very interesting. I agree that adding the clarifications in the rebuttal in the paper would strengthen the work. (Also the effort to make such changes would also be non-trivial lol). Given the responses, I encourage the authors explore tasks mentioned in the rebuttal (e.g. training deeper networks) with this new architectur for future works.

Authorsrebuttal2023-08-20

Response to Reviewer qcqa: Thanks for your endorsement!

Thanks for your valuable feedback. We are encouraged by your endorsements that our paper is non-trivial and very interesting. We will add the clarifications on the oversmoothing phenomenon in vision transformers and our results during the rebuttal, which show that reducing oversmoothing helps improve the model’s performance, into the revision. Following your suggestion, we will also include our new results for training NeuTRENO under adversarial examples and for out-of-distribution generalization, as well as for the incremental learning task, in the revised manuscript. We will further study the applications of NeuTRENO on these tasks and train deeper NeuTRENO in future works.

Area Chair n6M12023-08-18

Please read rebuttals

Dear reviewers, if you didn't already, please read the rebuttals ASAP and at least acknowledge them explicitly. Best, Area Chair

Reviewer h3ak2023-08-18

Thanks for the authors' response and clarifiaction. I keep my initial rating.

Authorsrebuttal2023-08-19

Additional Results on Oversmoothing and Empirical Performance

Dear reviewers, We would like to thank all reviewers again for your thoughtful reviews and valuable feedback. We have obtained additional empirical results to clarify the impact of oversmoothing on the model’s performance. We summarize our results below. In Figure 7 (Left) in the rebuttal PDF, our NeuTRENO BERT finetuned on the SQUAD question answering task [1] yields better accuracy than the baseline BERT finetuned on the same task (81.39 exact match score and 88.62 F1 score vs. 80.77 exact match score and 88.12 F1 score). In Figure 7 (Right) in the rebuttal PDF, our NeuTRENO SimCSE, after finetuned on the STS-12 semantic textual similarity task [2], gains a significant improvement over the baseline SimCSE [3], which is also finetuned on the same task (77.32% vs. 75.29% Spearman's correlation. Here, the higher correlation, the better). These results indicate that reducing cosine similarity between tokens in the trained transformer-based models leads to better empirical performance. We would also like to provide additional evidence that oversmoothing is indeed an issue for vision transformers (ViT). Indeed, the oversmoothing in ViT has been verified and investigated in existing works. In particular, [4] observes that the performance of ViT quickly saturates as more layers are added to the model. Moreover, experiments in [4] show that the 32-layer ViT underperforms the 24-layer ViT, indicating the difficulty of ViTs in gaining benefits from deeper architectures. The authors point out that oversmoothing results in this phenomenon by causing the token representations to become identical when the model grows deeper. Based on this observation, they propose a cross-head communication method that helps enhance the diversity of both token representations and attention matrices. Furthermore, it has been shown in [5] that the training of ViT models encounters instability with greater depths. [6] proposes that this instability arises from the oversmoothing, where token representations for patches within an image become progressively alike as the model's depth increases. In an effort to explain this issue, [7] finds out that self-attention acts as a low-pass filter, which smoothens the token representations in ViTs. This leads to the proposal of the FeatScale method [7], which regulates feature frequencies, whether low or high, to counteract the consequences of oversmoothing. Different from existing works, in our paper, we theoretically prove the oversmoothing phenomenon via the variational denoising framework that we develop. Our proposed NeuTRENO method helps mitigate the oversmoothing issue and improves the performance of the ViT baselines (DeiTs), as well as other baseline transformer-based models. We also empirically demonstrate that NeuTRENO not only reduces the cosine similarity between token representation (See Figure 1 in the main text and Figure 7) but also the redundancy in attention maps between layers (See Figure 5 in Appendix G2). Finally, we show that NeuTRENO is complementary to existing methods, including FeatScale [7] (See Table 1 and Table 7 in our paper). We would be happy to do any follow-up discussion or address any additional comments. **References** [1] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. “SQuAD: 100,000+ questions for machine comprehension of text”. EMNLP, 2016. [2] Eneko Agirre, et al. “SemEval-2012 task 6: A pilot on semantic textual similarity.” SemEval, 2012. [3] Gao, T., Yao, X., and Chen, D. "SimCSE: Simple contrastive learning of sentence embeddings". EMNLP, 2021. [4] Daquan Zhou, et al. “Deepvit: Towards deeper vision transformer”. arXiv, 2021. [5] Touvron Hugo, et al. "Going deeper with image transformers." ICCV, 2021. [6] Chengyue Gong, et al. “Vision transformers with patch diversification”. arXiv, 2021. [7] Wang Peihao, et al. "Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice." ICLR, 2022.

Reviewer k2Mg2023-08-21

Thanks for the authors' response. I have no further questions about this paper.

Authorsrebuttal2023-08-22

Thank You

Dear Reviewer qvvE, Thanks for your reviews of our paper. Since the discussion period between the authors and the reviewers was already over and we have not heard from you during this period, we would be grateful if the reviewer could let us know if all your questions are addressed to some extent. If you are satisfied with our answers, we hope that the reviewer will consider adjusting your score. Best regards, Authors

Authorsrebuttal2023-08-22

Thank You

Dear Reviewer YNWk, Thanks for your reviews of our paper. Since the discussion period between the authors and the reviewers was already over and we have not heard from you during this period, we would be grateful if the reviewer could let us know if all your questions are addressed to some extent. If you are satisfied with our answers, we hope that the reviewer will consider adjusting your score. Best regards, Authors

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC