WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization

As deep neural networks make their ways into different domains, their compute\nefficiency is becoming a first-order constraint. Deep quantization, which\nreduces the bitwidth of the operations (below 8 bits), offers a unique\nopportunity as it can reduce both the storage and compute requirements of the\nnetwork super-linearly. However, if not employed with diligence, this can lead\nto significant accuracy loss. Due to the strong inter-dependence between layers\nand exhibiting different characteristics across the same network, choosing an\noptimal bitwidth per layer granularity is not a straight forward. As such, deep\nquantization opens a large hyper-parameter space, the exploration of which is a\nmajor challenge. We propose a novel sinusoidal regularization, called SINAREQ,\nfor deep quantized training. Leveraging the sinusoidal properties, we seek to\nlearn multiple quantization parameterization in conjunction during\ngradient-based training process. Specifically, we learn (i) a per-layer\nquantization bitwidth along with (ii) a scale factor through learning the\nperiod of the sinusoidal function. At the same time, we exploit the\nperiodicity, differentiability, and the local convexity profile in sinusoidal\nfunctions to automatically propel (iii) network weights towards values\nquantized at levels that are jointly determined. We show how SINAREQ balance\ncompute efficiency and accuracy, and provide a heterogeneous bitwidth\nassignment for quantization of a large variety of deep networks (AlexNet,\nCIFAR-10, MobileNet, ResNet-18, ResNet-20, SVHN, and VGG-11) that virtually\npreserves the accuracy. Furthermore, we carry out experimentation using fixed\nhomogenous bitwidths with 3- to 5-bit assignment and show the versatility of\nSINAREQ in enhancing quantized training algorithms (DoReFa and WRPN) with about\n4.8% accuracy improvements on average, and then outperforming multiple\nstate-of-the-art techniques.\n