This paper is aimed at creating extremely small and fast convolutional neural\nnetworks (CNN) for the problem of facial expression recognition (FER) from\nfrontal face images. To this end, we employed the popular knowledge\ndistillation (KD) method and identified two major shortcomings with its use: 1)\na fine-grained grid search is needed for tuning the temperature hyperparameter\nand 2) to find the optimal size-accuracy balance, one needs to search for the\nfinal network size (or the compression rate). On the other hand, KD is proved\nto be useful for model compression for the FER problem, and we discovered that\nits effects gets more and more significant with the decreasing model size. In\naddition, we hypothesized that translation invariance achieved using\nmax-pooling layers would not be useful for the FER problem as the expressions\nare sensitive to small, pixel-wise changes around the eye and the mouth.\nHowever, we have found an intriguing improvement on generalization when\nmax-pooling is used. We conducted experiments on two widely-used FER datasets,\nCK+ and Oulu-CASIA. Our smallest model (MicroExpNet), obtained using knowledge\ndistillation, is less than 1MB in size and works at 1851 frames per second on\nan Intel i7 CPU. Despite being less accurate than the state-of-the-art,\nMicroExpNet still provides significant insights for designing a\nmicroarchitecture for the FER problem.\n