Neural gradients are near-lognormal: improved quantized and sparse training

While training can mostly be accelerated by reducing the time needed to\npropagate neural gradients back throughout the model, most previous works focus\non the quantization/pruning of weights and activations. These methods are often\nnot applicable to neural gradients, which have very different statistical\nproperties. Distinguished from weights and activations, we find that the\ndistribution of neural gradients is approximately lognormal. Considering this,\nwe suggest two closed-form analytical methods to reduce the computational and\nmemory burdens of neural gradients. The first method optimizes the\nfloating-point format and scale of the gradients. The second method accurately\nsets sparsity thresholds for gradient pruning. Each method achieves\nstate-of-the-art results on ImageNet. To the best of our knowledge, this paper\nis the first to (1) quantize the gradients to 6-bit floating-point formats, or\n(2) achieve up to 85% gradient sparsity -- in each case without accuracy\ndegradation. Reference implementation accompanies the paper.\n