Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation

Recent work on the lottery ticket hypothesis has produced highly sparse\nTransformers for NMT while maintaining BLEU. However, it is unclear how such\npruning techniques affect a model's learned representations. By probing\nTransformers with more and more low-magnitude weights pruned away, we find that\ncomplex semantic information is first to be degraded. Analysis of internal\nactivations reveals that higher layers diverge most over the course of pruning,\ngradually becoming less complex than their dense counterparts. Meanwhile, early\nlayers of sparse models begin to perform more encoding. Attention mechanisms\nremain remarkably consistent as sparsity increases.\n