ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training

Large-scale distributed training of Deep Neural Networks (DNNs) on\nstate-of-the-art platforms is expected to be severely communication\nconstrained. To overcome this limitation, numerous gradient compression\ntechniques have been proposed and have demonstrated high compression ratios.\nHowever, most existing methods do not scale well to large scale distributed\nsystems (due to gradient build-up) and/or fail to evaluate model fidelity (test\naccuracy) on large datasets. To mitigate these issues, we propose a new\ncompression technique, Scalable Sparsified Gradient Compression (ScaleCom),\nthat leverages similarity in the gradient distribution amongst learners to\nprovide significantly improved scalability. Using theoretical analysis, we show\nthat ScaleCom provides favorable convergence guarantees and is compatible with\ngradient all-reduce techniques. Furthermore, we experimentally demonstrate that\nScaleCom has small overheads, directly reduces gradient traffic and provides\nhigh compression rates (65-400X) and excellent scalability (up to 64 learners\nand 8-12X larger batch sizes over standard training) across a wide range of\napplications (image, language, and speech) without significant accuracy loss.\n