Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures

Recent years have seen a growing adoption of Transformer models such as BERT\nin Natural Language Processing and even in Computer Vision. However, due to\ntheir size, there has been limited adoption of such models within\nresource-constrained computing environments. This paper proposes novel pruning\nalgorithm to compress transformer models by eliminating redundant Attention\nHeads. We apply the A* search algorithm to obtain a pruned model with strict\naccuracy guarantees. Our results indicate that the method could eliminate as\nmuch as 40% of the attention heads in the BERT transformer model with no loss\nin accuracy.\n