Model Compression for IoT Applications in Industry 4.0 via Multiscale Knowledge Transfer

Abstract

Recently, Industry 4.0 has attracted much attention. It has close relations with the Internet of Things (IoT). On the other hand, convolutional neural networks (CNNs) have shown promising performance in many foundational services of the IoT applications. For the IoT applications with high-speed data streams and the requirement of time-sensitive actions, fast processing is demanded on small-scale platforms or even on IoT devices themselves. Therefore, it is inappropriate to employ cumbersome CNNs in IoT applications, making the study of model compression necessary. In knowledge transfer, it is common to employ a deep, well-trained network, called <italic>teacher</italic>, to guide a shallow, untrained network, called <italic>student</italic>, to have better performance. Previous works have made many attempts to transfer single-scale knowledge from <italic>teacher</italic> to <italic>student</italic>, leading to degradation of generalization ability. In this article, we introduce multiscale representations to knowledge transfer, which facilitates the generalization ability of <italic>student</italic>. We divide <italic>student</italic> and <italic>teacher</italic> into several stages. <italic>Student</italic> learns from multiscale knowledge provided by <italic>teacher</italic> at the end of each stage. Extensive experiments demonstrate the effectiveness of our proposed method both on image classification and on single image super-resolution. The huge performance gap between <italic>student</italic> and <italic>teacher</italic> is significantly narrowed down by our proposed method, making <italic>student</italic> suitable for IoT applications.