Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

The growing energy and performance costs of deep learning have driven the\ncommunity to reduce the size of neural networks by selectively pruning\ncomponents. Similarly to their biological counterparts, sparse networks\ngeneralize just as well, if not better than, the original dense networks.\nSparsity can reduce the memory footprint of regular networks to fit mobile\ndevices, as well as shorten training time for ever growing networks. In this\npaper, we survey prior work on sparsity in deep learning and provide an\nextensive tutorial of sparsification for both inference and training. We\ndescribe approaches to remove and add elements of neural networks, different\ntraining strategies to achieve model sparsity, and mechanisms to exploit\nsparsity in practice. Our work distills ideas from more than 300 research\npapers and provides guidance to practitioners who wish to utilize sparsity\ntoday, as well as to researchers whose goal is to push the frontier forward. We\ninclude the necessary background on mathematical methods in sparsification,\ndescribe phenomena such as early structure adaptation, the intricate relations\nbetween sparsity and the training process, and show techniques for achieving\nacceleration on real hardware. We also define a metric of pruned parameter\nefficiency that could serve as a baseline for comparison of different sparse\nnetworks. We close by speculating on how sparsity can improve future workloads\nand outline major open problems in the field.\n