LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification

Deep Neural Networks (DNNs) have become the de-facto standard in computer\nvision, as well as in many other pattern recognition tasks. A key drawback of\nDNNs is that the training phase can be very computationally expensive.\nOrganizations or individuals that cannot afford purchasing state-of-the-art\nhardware or tapping into cloud-hosted infrastructures may face a long waiting\ntime before the training completes or might not be able to train a model at\nall. Investigating novel ways to reduce the training time could be a potential\nsolution to alleviate this drawback, and thus enabling more rapid development\nof new algorithms and models. In this paper, we propose LightLayers, a method\nfor reducing the number of trainable parameters in deep neural networks (DNN).\nThe proposed LightLayers consists of LightDense andLightConv2D layer that are\nas efficient as regular Conv2D and Dense layers, but uses less parameters. We\nresort to Matrix Factorization to reduce the complexity of the DNN models\nresulting into lightweight DNNmodels that require less computational power,\nwithout much loss in the accuracy. We have tested LightLayers on MNIST, Fashion\nMNIST, CI-FAR 10, and CIFAR 100 datasets. Promising results are obtained for\nMNIST, Fashion MNIST, CIFAR-10 datasets whereas CIFAR 100 shows acceptable\nperformance by using fewer parameters.\n