Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks

The high energy cost of processing deep convolutional neural networks impedes their ubiquitous deployment in energy-constrained platforms such as embedded systems and IoT devices. This article introduces convolutional layers with pre-defined sparse 2D kernels that have support sets that repeat periodically within and across filters. Due to the efficient storage of our periodic sparse kernels, the parameter savings can translate into considerable improvements in energy efficiency due to reduced DRAM accesses, thus promising significant improvements in the trade-off between energy consumption and accuracy for both training and inference. To evaluate this approach, we performed experiments with two widely accepted datasets, CIFAR-10 and Tiny ImageNet in sparse variants of the ResNet18 and VGG16 architectures. Compared to baseline models, our proposed sparse variants require up to <inline-formula><tex-math notation="LaTeX">$\mathord {\sim }82\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mn>82</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="kundu-ieq1-2972520.gif"/></alternatives></inline-formula> fewer model parameters with <inline-formula><tex-math notation="LaTeX">$5.6\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>5</mml:mn><mml:mo>.</mml:mo><mml:mn>6</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="kundu-ieq2-2972520.gif"/></alternatives></inline-formula> fewer FLOPs with negligible loss in accuracy for ResNet18 on CIFAR-10. For VGG16 trained on Tiny ImageNet, our approach requires <inline-formula><tex-math notation="LaTeX">$5.8 \times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>5</mml:mn><mml:mo>.</mml:mo><mml:mn>8</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="kundu-ieq3-2972520.gif"/></alternatives></inline-formula> fewer FLOPs and up to <inline-formula><tex-math notation="LaTeX">$\mathord {\sim }83.3\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mn>83</mml:mn><mml:mo>.</mml:mo><mml:mn>3</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="kundu-ieq4-2972520.gif"/></alternatives></inline-formula> fewer model parameters with a drop in top-5 (top-1) accuracy of only 1.2% (<inline-formula><tex-math notation="LaTeX">$\mathord {\sim }2.1\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mn>2</mml:mn><mml:mo>.</mml:mo><mml:mn>1</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="kundu-ieq5-2972520.gif"/></alternatives></inline-formula>). We also compared the performance of our proposed architectures with that of ShuffleNet and MobileNetV2. Using similar hyperparameters and FLOPs, our ResNet18 variants yield an average accuracy improvement of <inline-formula><tex-math notation="LaTeX">$\mathord {\sim }2.8\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mn>2</mml:mn><mml:mo>.</mml:mo><mml:mn>8</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="kundu-ieq6-2972520.gif"/></alternatives></inline-formula>.