Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation

The deployment of widely used Transformer architecture is challenging because\nof heavy computation load and memory overhead during inference, especially when\nthe target device is limited in computational resources such as mobile or edge\ndevices. Quantization is an effective technique to address such challenges. Our\nanalysis shows that for a given number of quantization bits, each block of\nTransformer contributes to translation quality and inference computations in\ndifferent manners. Moreover, even inside an embedding block, each word presents\nvastly different contributions. Correspondingly, we propose a mixed precision\nquantization strategy to represent Transformer weights by an extremely low\nnumber of bits (e.g., under 3 bits). For example, for each word in an embedding\nblock, we assign different quantization bits based on statistical property. Our\nquantized Transformer model achieves 11.8$\\times$ smaller model size than the\nbaseline model, with less than -0.5 BLEU. We achieve 8.3$\\times$ reduction in\nrun-time memory footprints and 3.5$\\times$ speed up (Galaxy N10+) such that our\nproposed compression strategy enables efficient implementation for on-device\nNMT.\n