BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization

Mixed-precision quantization can potentially achieve the optimal tradeoff\nbetween performance and compression rate of deep neural networks, and thus,\nhave been widely investigated. However, it lacks a systematic method to\ndetermine the exact quantization scheme. Previous methods either examine only a\nsmall manually-designed search space or utilize a cumbersome neural\narchitecture search to explore the vast search space. These approaches cannot\nlead to an optimal quantization scheme efficiently. This work proposes\nbit-level sparsity quantization (BSQ) to tackle the mixed-precision\nquantization from a new angle of inducing bit-level sparsity. We consider each\nbit of quantized weights as an independent trainable variable and introduce a\ndifferentiable bit-sparsity regularizer. BSQ can induce all-zero bits across a\ngroup of weight elements and realize the dynamic precision reduction, leading\nto a mixed-precision quantization scheme of the original model. Our method\nenables the exploration of the full mixed-precision space with a single\ngradient-based optimization process, with only one hyperparameter to tradeoff\nthe performance and compression. BSQ achieves both higher accuracy and higher\nbit reduction on various model architectures on the CIFAR-10 and ImageNet\ndatasets comparing to previous methods.\n