When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute

Large language models have become increasingly difficult to train because of\nthe growing computation time and cost. In this work, we present SRU++, a\nhighly-efficient architecture that combines fast recurrence and attention for\nsequence modeling. SRU++ exhibits strong modeling capacity and training\nefficiency. On standard language modeling tasks such as Enwik8, Wiki-103 and\nBillion Word datasets, our model obtains better bits-per-character and\nperplexity while using 3x-10x less training cost compared to top-performing\nTransformer models. For instance, our model achieves a state-of-the-art result\non the Enwik8 dataset using 1.6 days of training on an 8-GPU machine. We\nfurther demonstrate that SRU++ requires minimal attention for near\nstate-of-the-art performance. Our results suggest jointly leveraging fast\nrecurrence with little attention as a promising direction for accelerating\nmodel training and inference.\n