When deep learning models on GPU can be accelerated by taking advantage of unstructured sparsity

This paper is focused on the improvement the efficiency of the sparse\nconvolutional neural networks (CNNs) layers on graphic processing units (GPU).\nThe Nvidia deep neural network (cuDnn) library provides the most effective\nimplementation of deep learning (DL) algorithms for GPUs. GPUs are one of the\nmost efficient and commonly used accelerators for deep learning computations.\nThe modern CNN models need megabytes of coefficients and needed millions MAC\noperations to perform convolution. One of the most common techniques for\ncompressing CNN models is weight pruning. There are two main types of pruning:\nstructural (based on removing whole weight channels) and non-structural\n(removing individual weights). The first enables much easier acceleration, but\nwith this type it is difficult to achieve a sparsity level and accuracy as high\nas that obtained with the second type. Non-structural pruning with retraining\ncan generate a matrix-weight up to $\\sim90\\%$ or more of sparsity in some deep\nCNN models. This work shows when is worth using a direct sparse operation to\nspeed-up the calculation of the convolution layers. The VGG-16, CNN-non-static\nand 1x1 layers from ResNet models were used as a benchmarks. In addition, we\npresent the impact of using reduced precision on time efficiency.\n