Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis

The remarkable success of transformers in sequence modeling tasks, spanning various applications in natural language processing and computer vision, is attributed to the critical role of self-attention. Similar to the development of most deep learning models, the construction of these attention mechanisms relies on heuristics and experience. In our work, we derive self-attention from kernel principal component analysis (kernel PCA) and show that self-attention projects its query vectors onto the principal component axes of its key matrix in a feature space. We then formulate the exact formula for the value matrix in self-attention, theoretically and empirically demonstrating that this value matrix captures the eigenvectors of the Gram matrix of the key vectors in self-attention. Leveraging our kernel PCA framework, we propose Attention with Robust Principal Components (RPC-Attention), a novel class of robust attention that is resilient to data contamination. We empirically demonstrate the advantages of RPC-Attention over softmax attention on the ImageNet-1K object classification, WikiText-103 language modeling, and ADE20K image segmentation task.

cs.LG cs.AI cs.CL cs.CV stat.ML Neural Networks and Applications Computer Science Mathematics

View graph Source

Paper

References (100)

01Revisiting Kernel Attention with Correlated Gaussian Process RepresentationL. Bui, Tho Tran Huu, Duy Dinh et al.2025 · Conference on Uncertainty in Artificial Intelligence · 13 citations In Library

02Equivariant Neural Functional Networks for TransformersHoang Tran-Viet, Thieu N. Vo, An Nguyen The et al.2024 · arXiv.org · 19 citations In Library

03Elliptical AttentionStefan K. Nielsen, L. Abdullaev, Rachel S. Y. Teo et al.2024 · Neural Information Processing Systems · 10 citations In Library

04A Primal-Dual Framework for Transformers and Neural NetworksT. Nguyen, T. Nguyen, Nhat Ho et al.2024 · International Conference on Learning Representations · 19 citations In Library

05PIDformer: Transformer Meets Control TheoryT. Nguyen, César A. Uribe, T. Nguyen et al.2024 · International Conference on Machine Learning · 13 citations In Library

06A mathematical perspective on TransformersBorjan Geshkovski, Cyril Letrouit, Yury Polyanskiy et al.2023 · Bulletin of the American Mathematical Society · 190 citations In Library

07Mitigating Over-smoothing in Transformers via Regularized Nonlocal FunctionalsT. Nguyen, T. Nguyen, R. Baraniuk2023 · Neural Information Processing Systems · 40 citations In Library

08Transformers as Support Vector MachinesDavoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis et al.2023 · arXiv.org · 71 citations In Library

09A Probabilistic Framework for Pruning Transformers Via a Finite Admixture of KeysT. Nguyen, T. Nguyen, L. Bui et al.2023 · IEEE International Conference on Acoustics, Speech, and Signal Processing · 7 citations

10Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal RepresentationYingyi Chen, Qinghua Tao, F. Tonin et al.2023 · Neural Information Processing Systems · 40 citations In Library

11The emergence of clusters in self-attention dynamicsBorjan Geshkovski, Cyril Letrouit, Yury Polyanskiy et al.2023 · Neural Information Processing Systems · 144 citations In Library

12Calibrating Transformers via Sparse Gaussian ProcessesWenlong Chen, Yingzhen Li2023 · International Conference on Learning Representations · 22 citations In Library

Scroll for more · 38 remaining

Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal RepresentationYingyi Chen, Qinghua Tao, F. Tonin et al.2023 · Neural Information Processing Systems · 40 citations In Library

A Primal-Dual Framework for Transformers and Neural NetworksTan M. Nguyen, Tam W. Nguyen, Nhat Ho et al.2024 · arXiv (Cornell University) · 19 citations In Library

The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-AttentionGiorgio Roffo2025 · arXiv.org · 1 citation In Library

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer trainingM. Saponati, Pascal Sager, Pau Vilimelis Aceituno et al.2025 · International Conference on Machine Learning · 9 citations In Library

Self-attention as the backbone: A survey on Vision TransformersAhmad Waseem, Pietro Ruiu, Seth Nixon et al.2026 · Computer Vision and Image Understanding · 1 citation In Library

Eigen Analysis of Self-Attention and its Reconstruction from Partial ComputationSrinadh Bhojanapalli, Ayan Chakrabarti, Himanshu Jain et al.2021 · arXiv (Cornell University) · 11 citations In Library

Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of KernelYao-Hung Hubert Tsai, Shaojie Bai, M. Yamada et al.2019 · Conference on Empirical Methods in Natural Language Processing · 333 citations In Library

Rethinking the Self-Attention in Vision TransformersKyungmin Kim, Bichen Wu, Xiaoliang Dai et al.2021 · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · 51 citations In Library

The CUR Decomposition of Self-attentionChong Wu, Maolin Che, Hong Yan2024 In Library

Patents near this paper

Artificial Intelligence System for Capturing Context by Dilated Self-AttentionUS 11,557,283 · MITSUBISHI ELECTRIC RESEARCH LABORATORIES, INC. · 2023 ATTENTION BASED SEQUENTIAL IMAGE PROCESSINGUS 10,671,918 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2020 ATTENTION BASED SEQUENTIAL IMAGE PROCESSINGUS 11,205,123 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2021 SELF-ATTENTION DEEP NEURAL NETWORK FOR ACTION RECOGNITION IN SURVEILLANCE VIDEOSUS 10,089,556 · KONICA MINOLTA LABORATORY U.S.A., INC. · 2018 STRUCTURE-PRESERVING ATTENTION MECHANISM IN SEQUENCE-TO-SEQUENCE NEURAL MODELSUS 11,556,782 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2023

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview

Reviewer rAS85/10 · confidence 5/52024-07-13

Summary

This paper introduces a new method, Kernel PCA Attention (RPC-Attention), designed to enhance the self-attention mechanisms in transformers widely used in sequence modeling for both natural language processing and computer vision. The method applies principal component analysis to derive self-attention kernels, which helps in projecting key vectors into a feature space that prioritizes essential components, thereby optimizing task performance. The effectiveness of RPC-Attention is empirically validated through improved results in object classification, language modeling on WikiText-103, and image segmentation on the ADE20K dataset.

Strengths

1. The paper reveals the intrinsic connection between the self-attention mechanism and PCA/kernel PCA, providing a thorough mathematical explanation。 2. The derivation part is convincing and interesting.

Weaknesses

1. My main concern is about the practicality of the novel RPC-Attention mechanism. In your experiments, RPC-SymViT can only scale to a small size, and it does not show a significant performance advantage compared to ViT-Small. 2. Additionally, RPC-SymViT does not have an advantage in computational speed; for example, in Table 11, the inference time for RPC-SymViT (2iter/layer) is 70% higher than that of ViT. As it stands, the authors have not convinced me to abandon the traditional ViT in favor of RPC-SymViT. 3. I am also concerned about the scalability of this model, which can outperform ViT by ~1% on ImageNet at the tiny size, but the performance becomes very close at the small size. What happens if the model is scaled up even more? Do the assumptions in the paper still hold when the feature dimension is higher?

Questions

1. Did the authors try larger models for RPC-Attention?

Rating

Confidence

Soundness

Presentation

Contribution

Limitations

The authors have discussed their limitations; Scalability might be a vital limitation to the models proposed in this paper.

Authorsrebuttal2024-08-11

Any Questions from Reviewer rAS8 on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful reviews and valuable feedback. We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal. We would be happy to do any follow-up discussion or address any additional comments.

Authorsrebuttal2024-08-12

Another Call for Your Response

Dear Reviewer rAS8, We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Author-Reviewer Discussion period ends in less than two days from this comment, i.e., 11:59 pm AoE on August 13th. We are happy to answer any further questions you may have before then, but we will be unable to respond after that time. If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments! Sincerely, Authors

Reviewer rAS82024-08-13

Response to authors' rebutall

Thanks to the authors for their time and efforts in the rebuttal. The comparison of inference speed is now clear to me, but my concern of scalability remains. As the authors pointed out, "Our assumptions in Remark 2, do not hold when the feature dimension is higher than the sequence length as the number of principal components cannot exceed the rank of the covariance matrix." Does it mean that your base-sized model can only process sequences longer than 768? If so, this should be a strong limitation to your model. The claim " in practical tasks, the feature dimension is very often smaller than the sequence length" actually does not hold in many cases. For example, the most commonly compared baseline of ViT is a ViT-Base with 768-d features but only 197 tokens in the sequence (for 224 input and 16 patch size).

Authorsrebuttal2024-08-13

Response to the Reviewer's Concern about the Scalability of Our Model

Thanks for your reply. We respectfully disagree with the reviewer that scalability is a concern about our work. Please allow us to explain why scalability is not an issue below. In practice and also in our experiments, transformers use multihead attention with $H$ heads. The feature dimension mentioned in our assumptions in Remark 2 is the feature dimension at each head, which is $D/H$, where $D=768$ and $H=12$ in ViT-Base. Thus, in this case, the feature dimension ($768/12=64$) is smaller than the sequence length ($197$). For ViT-Base, our RPC-Attention model works with sequences longer than $64$. Furthermore, we can always reduce this number ($64$) by introducing more heads, but tasks with sequence lengths less than $64$ are not very practical. The same holds for ViT-Tiny, which has $D=192$ and $H=3$. As a result, we believe our statement that "in practical tasks, the feature dimension is very often smaller than the sequence length" is correct, and our assumptions in Remark 2 are still valid and should not raise any concern about the scalability of our RPC-Attention. Below, we provide a table that contains the feature dimensions, the number of heads, and the number of feature dimensions per head of each ViT model. | Model | Total Feature Dimensions ($D$) | Number of Heads ($H$) |Feature Dimensions per Head ($D/H$) | | -------- | -------- | -------- |-------- | | ViT-Tiny-Patch16 | 192 | 3 | 64 | | ViT-Small-Patch16 | 384 | 6 | 64 | | ViT-Base-Patch16 | 768 | 12 | 64 | | ViT-Large-Patch16 | 1024 | 16 | 64 | | ViT-Huge-Patch16 | 1280 | 16 | 80 |

Reviewer rAS82024-08-13

my concerns are addressed

Thanks for the authors‘ detailed clarification. My concern is now addressed and I raise my score to 5.

Reviewer 85Cd7/10 · confidence 3/52024-07-14

Summary

This paper studies the self-attention mechanism. The authors recover self-attention from kernel principle component analysis (kernel PCA). The authors empirically verify that the projection error is being minimized during training, suggesting that the transformer model is actually training to perform PCA during training. Also, the authors empirically confirm that the value matrix capture the eigenvectors of the Gram matrix. Lastly, using the analysis framework, the authors propose a robust attention based on principal component pursuit that is robust to input corruption and attacks.

Strengths

1. Viewing self-attention from kernel PCA perspective is very interesting. 2. The kernel PCA analysis of self-attention is supported with empirical evidence. 3. The analysis not only provides the theoretical understanding of self-attention, but also helps guiding the design of a new class of robust self-attention that is robust to input corruption and attacks.

Weaknesses

1. The RPC-Attention uses an iterative algorithm PAP to solve the convex formulation of principal component pursuit. How is back-propagation via PAP algorithm calculated? Does it lead to any instability? 2. The appendix E8 shows the runtime and memory of RPC-Attention is comparable to baseline despite using an iterative algorithm. I am curious about why it is not much slower. Is there any analysis? 3. I noticed that the baseline SymViT that uses symmetric attention has significantly worse accuracy compared to vanilla ViT on clean ImageNet (for tiny model, SymViT accuracy is 70.44, while vanilla ViT is around 75). I am wondering what would be the reason. If this is caused by the use of symmetric attention, is there a way to adopt RPC-Attention for vanilla ViT? 4. Given that RPC-Attention is more robust and as efficient as baseline, why the majority of experiments only apply it on the first few layers or the first layer, but not apply RPC-Attention on all layers (for example, like 6iter/all-layer).

Questions

see weakness section.

Rating

Confidence

Soundness

Presentation

Contribution

Limitations

The authors discussed a limitation about the potential efficiency issue of the proposed algorithm.

Authorsrebuttal2024-08-11

Any Questions from Reviewer 85Cd on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful reviews and valuable feedback. Regarding applying RPC for more iterations at all layers, as we mentioned, this results in higher computational overhead and may only lead to small improvements. We conducted an additional experiment with 4 iterations in all layers (RPC-SymViT (4iter/all-layer)) and provide the results in Table 1 and 2 below. We observe that RPC-SymViT (4iter/all-layer) indeed does outperform RPC-SymViT (2iter/all-layer) on almost all robustness benchmarks but still does not perform as well as RPC-SymViT(5/6iter/layer1). Interestingly, while RPC-SymViT (4iter/all-layer) performs significantly better than the baseline and RPC-SymViT(4/5/6iter/layer1) on PGD and FGSM attacked ImageNet-1K, RPC-SymViT (2iter/all-layer) still has the best accuracy. Table 1: Top-1, top-5 accuracy (%) , mean corruption error (mCE), and area under the precision-recall curve (AUPR) of RPC-SymViT and SymViT on clean ImageNet-1K data and popular standard robustness benchmarks for image classification. RPC-SymViT ($n$iter/layer1) applies $n$ PAP iterations only at the first layer. RPC-SymViT ($n$iter/all-layer) applies $n$ PAP iterations at all layers. | Model | IN-1K Top-1 ↑ | IN-1K Top-5 ↑ | IN-R Top-1 ↑ | IN-A Top-1 ↑ | IN-C Top-1 ↑ | IN-C mCE ↓ | IN-O AUPR ↑ | |----------------------------|---------------|---------------|--------------|--------------|--------------|------------|-------------| | SymViT (baseline) | 70.44 | 90.17 | 28.98 | 6.51 | 41.45 | 74.75 | 17.43 | | RPC-SymViT (4iter/layer1) | 70.94 | 90.47 | 29.99 | 6.96 | 42.35 | 73.58 | 19.32 | | RPC-SymViT (5iter/layer1) | 71.31 | 90.59 | **30.28** | 7.27 | 42.43 | 73.43 | **20.35** | | RPC-SymViT (6iter/layer1) | **71.49** | **90.68** | 30.03 | 7.33 | **42.76** | **73.03** | 20.29 | | RPC-SymViT (2iter/all-layer)| 70.59 | 90.15 | 29.23 | **7.55** | 41.64 | 74.52 | 19.18 | | RPC-SymViT (4iter/all-layer)| 71.17 | 90.61 | 30.09 | 7.43 | 42.55 | 73.35 | 19.43| Table 2: Top-1, top-5 accuracy (%) on PGD and FGSM attacked ImageNet-1K validation data with the highest perturbation. RPC-SymViT ($n$iter/layer1) applies $n$ PAP iterations only at the first layer. RPC-SymViT ($n$iter/all-layer) applies $n$ PAP iterations at all layers. | Model | PGD Top-1 ↑ | PGD Top-5 ↑ | FGSM Top-1 ↑ | FGSM Top-5 ↑ | |---------------|-------------|-------------|--------------|--------------| | SymViT-Tiny (baseline) | 4.98 | 10.41 | 23.38 | 53.82 | | RPC-SymViT (4iter/layer1) | 5.15 | 11.20 |26.62 | 56.87 | | RPC-SymViT (5iter/layer1) | 5.11 | 11.13 | 26.75 | 57.19 | | RPC-SymViT (6iter/layer1) | 5.20 |11.34 | 27.22 | 57.55| | RPC-SymViT (2iter/all-layer) | **6.12** | **13.24** | **29.20** | **59.63** | | RPC-SymViT (4iter/all-layer) | 5.46 | 12.17 | 27.99 | 59.01 | We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal. We would be happy to do any follow-up discussion or address any additional comments.

Authorsrebuttal2024-08-12

Discussion Deadline is in Less than Two Days

Dear Reviewer 85Cd, We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Author-Reviewer Discussion period ends in less than two days from this comment, i.e., 11:59 pm AoE on August 13th. We are happy to answer any further questions you may have before then, but we will be unable to respond after that time. If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your confidence score (3 currently) would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments! Sincerely, Authors

Reviewer hhSg7/10 · confidence 2/52024-07-26

Summary

The paper proposes a new perspective for understanding the underlying operation learned by scaled dot-product self-attention from a kernel PCA perspective. The paper recovers the self-attention formulation starting from PCA over feature projections of data points. In particular, the authors show that different parameterizations for the value matrix, while mathematically equivalent, can lead to different optimization problems and therefore different outcomes. Based on the new perspective provided in the paper, the authors propose RPC-Attention, a method to address one of the known vulnerabilities of PCA with corrupted data. The authors provide several experiments to demonstrate the effectiveness of RPC-Attention.

Strengths

1. The paper provides a very interesting and important perspective for understanding self-attention from the lens of Kernel PCA. Given the ubiquity and effectiveness of the transformer architecture, such insights are extremely valuable. 2. The paper is very well written, and the derivations are clear and well-explained. 3. In addition to clearly explaining the new perspective for understanding self-attention, the authors went ahead and addressed one of the weaknesses self-attention might suffer from given this new understanding. This is very valuable and gives a good example of how new perspectives can pave the way to key improvements in the underlying mechanisms. 3. The experimental results reflect the effectiveness of the proposed RPC-Attention method in improving the robustness of the models against adversarial attaches and corrupted data.

Weaknesses

1. [minor] The experiments use relatively small-scale models and short training runs (e.g. 50 epochs on Imagenet). While not a core issue given the objective of the paper, it would be nice to see results in more standardized settings (e.g. ViT-Base with 300 epochs)

Questions

1. Given the new perspective provided in the paper, do the authors have any insights into how to reduce the complexity of self-attention while retaining the same quality?

Rating

Confidence

Soundness

Presentation

Contribution

Limitations

Yes

Authorsrebuttal2024-08-11

Any Questions from Reviewer hhSg on Our Rebuttal?

Authorsrebuttal2024-08-12

Discussion Deadline is in Less than Two Days

Dear Reviewer hhSg, We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Author-Reviewer Discussion period ends in less than two days from this comment, i.e., 11:59 pm AoE on August 13th. We are happy to answer any further questions you may have before then, but we will be unable to respond after that time. If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your confidence score (2 currently) would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments! Sincerely, Authors

Reviewer hhSg2024-08-13

I thank the authors for the rebuttal. The rebuttal has addressed my main questions/concerns. The additional insights about possible ways to reduce the attention complexity are much appreciated. Therefore, I would like to keep my score of Accept.

Reviewer 8tie6/10 · confidence 3/52024-07-29

Summary

The paper derives self-attention from kernel principal component analysis (kernel PCA), showing that the attention outputs are projections of query vectors onto the principal component axes of the key matrix in a feature space. Using this kernel PCA framework, the authors propose Attention with Robust Principal Components (RPC-Attention), a robust attention that is resilient to data contamination, and demonstrate its effectiveness on ImageNet-1K object classification, WikiText-103 language14modeling, and ADE20K image segmentation task.

Strengths

1. This paper offers a novel perspective on attention mechanisms, proposing that attention outputs are projections of query vectors onto the principal component axes of the key matrix in a feature space. 2. The authors provide a detailed mathematical proof to support this claim. 3. The paper's experimental evaluation is comprehensive, demonstrating the applicability of the proposed RPC-attention mechanism across both vision and language tasks. 4. The writing is clear and well-organized, making the paper a pleasure to read.

Weaknesses

1. The results in Table 1 show that the best performance on different tasks is achieved with varying settings. For example, RPC achieves the best performance with 6 iterations/layer 1 on IN-1K, while RPC achieves the best performance with 2 iterations/all layers on IN-A. Moreover, for IN-1K top-5, 2 iterations/all layers even performs worse than the baseline attention. This suggests that more RPC attention is not always better for different tasks, and sometimes even performs worse than the baseline. Can authors provide an explanation for this phenomenon? 2. As shown in Appendix Table 11, the computational cost of RPC attention increases substantially with the number of iterations/layers. This raises concerns about its practical applicability when extended to multi-layer settings. Specifically, it is unclear whether the performance gains will be sufficient to justify the increased computational cost. This directly affects the method's usability in applications.

Questions

For language tasks, ppl may not accurately reflect the final performance. Have the authors conducted experiments on other downstream natural language understanding tasks to evaluate the effectiveness? Will RPA be applicable to pre-trained language models?

Rating

Confidence

Soundness

Presentation

Contribution

Limitations

Please refer to questions.

Authorsrebuttal2024-08-11

Any Questions from Reviewer 8tie on Our Rebuttal?

Authorsrebuttal2024-08-12

Another Call for Your Response

Dear Reviewer 8tie, We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Author-Reviewer Discussion period ends in less than two days from this comment, i.e., 11:59 pm AoE on August 13th. We are happy to answer any further questions you may have before then, but we will be unable to respond after that time. If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments! Sincerely, Authors

Authorsrebuttal2024-08-13

Additional Results on Downstream Natural Language Understanding Tasks (in addition to those in Table 4 in the attached PDF)

To further address your concerns, we have conducted an additional experiment on another downstream natural language understanding task in addition to those in Table 4 in the attached PDF. We use a pre-trained transformer language model from the NAACL 2019 tutorial on "Transfer Learning in Natural Language Processing" and finetune the model on the 5-class sentiment classification task, Stanford Sentiment Treebank (SST-5). We implement RPC-Attention during finetuning only (RPC-LM) and compare the results with the baseline model (Baseline-LM) on SST-5 in Table 1 below. As can be seen from the table, our RPC-Attention is applicable to pre-trained language models and performs significantly better than the baseline. Hence, RPC-Attention is highly effective in downstream natural language understanding tasks. Table 1: Validation and test accuracy of RPC-Attention implemented in a pre-trained transformer language model during finetuning versus the baseline transformer language model. We finetune both models on the 5-class Stanford Sentiment Treebank (SST-5) task. | Model | Validation Accuracy (%) | Test Accuracy (%) | | -------- | -------- | -------- | | Baseline-LM | 46.51 | 49.23 | |RPC-LM | **48.68** | **50.36** | We are happy to answer any further questions you may have. If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Reviewer 8tie2024-08-13

Thank you to the authors for addressing my concerns. I've increased my score to 6.

Reviewer 9vVz5/10 · confidence 5/52024-07-31

Summary

This work introduces a method for understanding attention mechanisms using kernel principal component analysis (kernel PCA). From this perspective, self-attention is viewed as projecting the query vectors onto the principal components of the key matrix. Building on this framework, this work develops a new robust attention mechanism, termed Attention with Robust Principal Components (RPC-Attention). RPC-Attention formulates the attention layer as an optimization problem of low-rank matrix recovery under sparse noise conditions, through Robust Principal Component Analysis. It utilizes the Alternating Direction Method of Multipliers (ADMM) algorithm to perform the forward pass of RPC-Attention. The proposed RPC-Attention layer converges within 4 to 6 iterations during the forward pass and shows robustness against data corruption and adversarial attacks.

Strengths

1. Soundness: The analysis of the attention mechanism presented in the manuscript is generally sound. The development of RPC-Attention is well-motivated by the formulation of standard self-attention. 2. Writing: The paper is well-motivated and written with clarity. 3. Evaluations: The evaluations encompass a wide range of common tasks in computer vision and language modeling. These evaluations support the claim of the proposed RPC-Attention's robustness against data corruption and adversarial attacks.

Weaknesses

1. Related works. The manuscript omits several relevant works. References [1,2] are particularly pertinent given the connection between attention mechanisms and decomposition methods for recovering clean signal subspaces discussed in this work. Additionally, RPC-Attention, as an iterative optimization layer, also links to OptNet [3] and Deep Equilibrium Models (DEQs) [4]. [1] Is Attention Better Than Matrix Decomposition? [2] Graph Neural Networks Inspired by Classical Iterative Algorithms [3] OptNet: Differentiable Optimization as a Layer in Neural Networks [4] Deep Equilibrium Models 2. Speed. A major deficiency of RPC-Attention is that it adds cost to the inference speed.

Questions

1. Gradient Flow: Iterative optimization layers often face issues with unstable gradient flow. How does RPC-Attention handle gradient propagation through the layer? Does it utilize implicit differentiation [3,4], inexact gradient methods [1], or Backpropagation Through Time (BPTT)? Specifically, are there any instabilities when differentiating through the softmax operator? 2. Convergence: How does the RPC-Attention layer converge in the forward pass when using 4 or 6 iters? What if training the layer using 4 iterations and performing inference with 6 iterations? Would improved convergence at test time enhance the results?

Rating

Confidence

Soundness

Presentation

Contribution

Limitations

The authors have discussed the limitations in the conclusion section. Given the connection between attention mechanisms and Graph Neural Networks (GNNs), it would be interesting to see further exploration of RPC-Attention applied to graph data.

Authorsrebuttal2024-08-11

Any Questions from Reviewer 9vVz on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful reviews and valuable feedback. Regarding your suggestion on further exploration of RPC-Attention applied to graph data, we have implemented RPC-Attention in a GAT model to replace Edge Attention. We train the baseline GAT model and RPC-GAT on the Cora dataset with the same settings as in [Veličković, 2017] for 10 seeds and report the average, best validation accuracy, as well as their standard deviation, in Table 1 below. Our RPC-GAT outperforms baseline GAT by 0.76%. Table 1: RPC-GAT vs. GAT on the Cora dataset. | Model | Val Acc (%) | | -------- | -------- | | GAT (baseline) | 81.28 $\pm$ 0.3% | | RPC-GAT | **82.04 $\pm$ 0.5%** | We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal. We would be happy to do any follow-up discussion or address any additional comments. **References** Veličković, Petar, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. "Graph attention networks." arXiv preprint arXiv:1710.10903 (2017).

Authorsrebuttal2024-08-12

Another Call for Your Response

Dear Reviewer 9vVz, We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Author-Reviewer Discussion period ends in less than two days from this comment, i.e., (11:59 pm AoE on August 13th). We are happy to answer any further questions you may have before then, but we will be unable to respond after that time. If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments! Sincerely, Authors

Reviewer 9vVz2024-08-13

The demonstration of robustness gain from RPC-Attention is sound, considering RPCA is designed for arbitrarily large sparse corruptions. I have raised my score to 5.

Authorsrebuttal2024-08-13

Thanks for your endorsement!

Thanks for your response, and we appreciate your endorsement. Following your suggestions, we will include comparisons with relevant works, clarification on the backward pass, and experiments with graph data discussed in our rebuttal in our revision.

Authorsrebuttal2024-08-12

Any Questions about Our Rebuttal

Dear Reviewers, We would like to thank you very much for your feedback, and we hope that our response addresses your previous concerns. In case you have not responded to our rebuttal so far, please feel free to let us know if you have any further comments on our work as the discussion period is expected to end soon (11:59pm AoE on August 13th). We would be more than happy to address any additional concerns from you. Thank you again for spending time on the paper, we really appreciate that! Best regards, The Authors

Reviewer 9vVz2024-08-13

I appreciate the authors' efforts to address my concerns. Particularly, 1. Comparisons with relevant works differentiate RPC-Attention from existing approaches, which are beneficial for broader readers. 2. Clarify the backward pass definition, easing reproduction and improving the clarity. 3. I also acknowledge the efforts on graph data. I have no concerns about acceptance.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

Source of record

arXiv

Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis

Paper

References (100)

Similar papers

Patents near this paper

Peer review

Welcome.

Patents near this paper