A contextual analysis of multi-layer perceptron models in classifying hand-written digits and letters: limited resources

Classifying hand-written digits and letters has taken a big leap with the\nintroduction of ConvNets. However, on very constrained hardware the time\nnecessary to train such models would be high. Our main contribution is twofold.\nFirst, we extensively test an end-to-end vanilla neural network (MLP) approach\nin pure numpy without any pre-processing or feature extraction done beforehand.\nSecond, we show that basic data mining operations can significantly improve the\nperformance of the models in terms of computational time, without sacrificing\nmuch accuracy. We illustrate our claims on a simpler variant of the Extended\nMNIST dataset, called Balanced EMNIST dataset. Our experiments show that,\nwithout any data mining, we get increased generalization performance when using\nmore hidden layers and regularization techniques, the best model achieving\n84.83% accuracy on a test dataset. Using dimensionality reduction done by PCA\nwe were able to increase that figure to 85.08% with only 10% of the original\nfeature space, reducing the memory size needed by 64%. Finally, adding methods\nto remove possibly harmful training samples like deviation from the mean helped\nus to still achieve over 84% test accuracy but with only 32.8% of the original\nmemory size for the training set. This compares favorably to the majority of\nliterature results obtained through similar architectures. Although this\napproach gets outshined by state-of-the-art models, it does scale to some\n(AlexNet, VGGNet) trained on 50% of the same dataset.\n