Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling

While Transformer-based models have shown impressive language modeling\nperformance, the large computation cost is often prohibitive for practical use.\nAttention head pruning, which removes unnecessary attention heads in the\nmultihead attention, is a promising technique to solve this problem. However,\nit does not evenly reduce the overall load because the heavy feedforward module\nis not affected by head pruning. In this paper, we apply layer-wise attention\nhead pruning on All-attention Transformer so that the entire computation and\nthe number of parameters can be reduced proportionally to the number of pruned\nheads. While the architecture has the potential to fully utilize head pruning,\nwe propose three training methods that are especially helpful to minimize\nperformance degradation and stabilize the pruning process. Our pruned model\nshows consistently lower perplexity within a comparable parameter size than\nTransformer-XL on WikiText-103 language modeling benchmark.\n