Summary
The paper introduces the first attempt at achieving 1-bit fully quantized training (FQT), aka quantizing weights, activation and gradients at 1-bit.
They first provided a theoretical analysis demonstrating that under the assumption of a bounded gradient, the convergence rate for a convex loss is influenced by the variance of the gradient. This informs their hypothesis that the Adam optimizer is better suited for FTQ compared to SGF.
They also show that the quantized gradient variance increases quadratically as the bitwidth decreases, suggesting the variance of 1-bit per-sample gradient quantization is too large for training. To address this, they propose the average 1-bit per-group quantization that randomly prunes samples proportionally to their dynamic range. They also propose a similar scheme for weight gradient, called Sample Channel Joint Quantization, that applies the same idea to groups of channels.
They perform various fine-tuning experiments and consistently show they outperform vanilla per-sample quantization and are within an average 5.5% drop for ResNet18 and 5% for VVGNet-16 on a collection of downstream tasks. They also make the first attempt at FTQ from scratch on CIFAR-10/100 and ImageNet, but the results there are way below the QAT. They also test their implementation on Hygon and Raspberry Pi, showing consistent training speedups.
Strengths
A very decent effort by the authors to provide a framework for 1-bit FTQ. To my knowledge, they are the first to undertake that daunting task. Even if the results are disappointing in terms of accuracy, it can be the basis of future work that will further close the gap. I summarise their strengths:
* The first to tackle the difficult task of 1-bit FTQ, even if their contributions are limited IMO.
* Solid presentation and easy-to-read paper.
* A theoretical motivation for their suggested methods and of the optimizers.
* Extensive experimentation covering many tasks, including image classification, object detection and NLP
* A very detailed appendix including ablations studies and implementation details for reproducibility. I specifically like that they consider the generalizability of their approach by applying it to more advanced binary models and showing an improvement even at higher bits.
Weaknesses
Whereas the paper is very well presented and tackles a novel and difficult however, its contribution is quite limited as it mainly builds on theory and methods already developed. Chen 2020 first introduces the relationship between quantized gradient and gradient range found in section 4.2. They motivate their per-sample quantization (PSQ) based on this and introduce the Block Householder Quantizer (BHQ). The authors compare their method to PSQ in their experiments but not to BHQ, which follows the same idea as their average 1-bit quantization.
I also find their conclusion about the convergence properties problematic because there is little evidence to suggest that the loss function of neural networks is convex. In addition, they compare vanilla SGD to Adam without considering momentum SGD first. Adam does indeed have great properties for training networks, but it’s hardly surprising that SGD without momentum fails in the high variance regime of low-bit quantization. This is why momentum is introduced to act as a low-pass filter.
The authors introduce their FTQ method mainly for on-device fine-tuning. However, they do not discuss certain limitations regarding data movement and memory restrictions on edge devices. According to the appendix, they use large batches of size 64 for on-device fine-tuning (which may not be realistic for real edge applications). In addition, the dequantization operation in the step of algorithm 1 would require storing the large high-precision tensors (assuming this is float32) in memory to first calculate the dynamic ranges and perform the pruning/quantization later. At such low precision, the training would most likely become memory-bound. It would be useful to see a profiling of training on the device to better understand the impact of the high-precision operations
Questions
* Fig. 1 what is the quantization scheme for the gradient?
* Tables 2 & 3 are very small and hard to read. Can you please expand them a bit?
* It would be interesting to see a comparison of their method to the Per-Sample Quantizer combined with the Block Householder Quantizer for activation gradients.
* A profiling of the training process on either of the edge devices, focusing on the compute time spent on the high-precision operations, e.g. Batch-Norm gradient estimation, and the memory bandwidth of activation gradients.
* Using the symbol of a Hessian $\mathbf{H}$ to denote activations in section 3 is a bit confusing (but this is just a remark)
* The equations in section 3.1 have no numbering. I believe that a quantization function must be applied in line 149 for the equations to hold. The product of two quantize tenses: $\hat{\nabla}_{\mathbf{H}^{(l)}} \bar{\mathbf{\Theta}}^{(l)^T}$ will not any more be in the same precision as the two inputs. Shouldn't it be instead
$$Q_g(\hat{\nabla}_{\mathbf{H}^{(l)}} \bar{\mathbf{\Theta}}^{(l)^T})$$