Variant Parallelism: Lightweight Deep Convolutional Models for Distributed Inference on IoT Devices

Two major techniques are commonly used to meet real-time inference limitations when distributing models across resource-constrained IoT devices: 1) model parallelism (MP) and 2) class parallelism (CP). In MP, transmitting bulky intermediate data (orders of magnitude larger than input) between devices imposes huge communication overhead. Although CP solves this problem, it has limitations on the number of submodels. In addition, both solutions are fault intolerant, an issue when deployed on edge devices. We propose variant parallelism (VP), an ensemble-based deep learning distribution method where different variants of a main model are generated and can be deployed on separate machines. We design a family of lighter models around the original model, and train them simultaneously to improve accuracy over single models. Our experimental results on six common mid-sized object recognition data sets demonstrate that our models can have <inline-formula> <tex-math notation="LaTeX">$5.8\times $ </tex-math></inline-formula>–<inline-formula> <tex-math notation="LaTeX">$7.1\times $ </tex-math></inline-formula> fewer parameters, <inline-formula> <tex-math notation="LaTeX">$4.3\times $ </tex-math></inline-formula>–<inline-formula> <tex-math notation="LaTeX">$31\times $ </tex-math></inline-formula> fewer multiply accumulations (MACs), and <inline-formula> <tex-math notation="LaTeX">$2.5\times $ </tex-math></inline-formula>–<inline-formula> <tex-math notation="LaTeX">$13.2\times $ </tex-math></inline-formula> less response time on atomic inputs compared to MobileNetV2 while achieving comparable or higher accuracy. Our technique easily generates several variants of the base architecture. Each variant returns only <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {2k}$ </tex-math></inline-formula> outputs <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {1 \leq k \leq ({\#classes}/{2})}$ </tex-math></inline-formula>, representing <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {Top{-} k}$ </tex-math></inline-formula> classes, instead of tons of floating point values required in MP. Since each variant provides a full-class prediction, our approach maintains higher availability compared with MP and CP in presence of failure.