On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks

We examine how recently documented, fundamental phenomena in deep learning\nmodels subject to pruning are affected by changes in the pruning procedure.\nSpecifically, we analyze differences in the connectivity structure and learning\ndynamics of pruned models found through a set of common iterative pruning\ntechniques, to address questions of uniqueness of trainable, high-sparsity\nsub-networks, and their dependence on the chosen pruning method. In\nconvolutional layers, we document the emergence of structure induced by\nmagnitude-based unstructured pruning in conjunction with weight rewinding that\nresembles the effects of structured pruning. We also show empirical evidence\nthat weight stability can be automatically achieved through apposite pruning\ntechniques.\n