Revisiting Linformer with a modified self-attention with linear complexity

Although Transformer models such as Google's BERT and OpenAI's GPT-3 are\nsuccessful in many natural language processing tasks, training and deploying\nthese models are costly and inefficient.Even if pre-trained models are used,\ndeploying these models still remained a challenge due to their large size.\nApart from deployment, these models take higher time during inference\nrestricting user-friendliness. The main bottleneck is self-attention which uses\nquadratic time and space with respect to the sequence length. In order to\nreduce the quadratic time complexity of the self-attention mechanism, Linformer\nby Facebook's AI research team was introduced where they showed that the\nself-attention mechanism can be approximated by a low-rank matrix and\nexploiting this finding, a new method for self-attention with linear time and\nspace complexity was proposed by them. In the Linformer, the time complexity\ndepends on the projection mapping dimension which acts as a hyperparameter and\naffects the performance of the model, tuning this hyperparameter can be\ntime-consuming. In this paper, I proposed an alternative method for\nself-attention with linear complexity in time and space and is independent of\nthe projection mapping dimension. Since this method works for long sequences\nthis can be used for images as well as audios.\n