Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

We present a novel network pruning algorithm called Dynamic Sparse Training\nthat can jointly find the optimal network parameters and sparse network\nstructure in a unified optimization process with trainable pruning thresholds.\nThese thresholds can have fine-grained layer-wise adjustments dynamically via\nbackpropagation. We demonstrate that our dynamic sparse training algorithm can\neasily train very sparse neural network models with little performance loss\nusing the same number of training epochs as dense models. Dynamic Sparse\nTraining achieves the state of the art performance compared with other sparse\ntraining algorithms on various network architectures. Additionally, we have\nseveral surprising observations that provide strong evidence for the\neffectiveness and efficiency of our algorithm. These observations reveal the\nunderlying problems of traditional three-stage pruning algorithms and present\nthe potential guidance provided by our algorithm to the design of more compact\nnetwork architectures.\n