Preserved central model for faster bidirectional compression in distributed settings

We develop a new approach to tackle communication constraints in a\ndistributed learning problem with a central server. We propose and analyze a\nnew algorithm that performs bidirectional compression and achieves the same\nconvergence rate as algorithms using only uplink (from the local workers to the\ncentral server) compression. To obtain this improvement, we design MCM, an\nalgorithm such that the downlink compression only impacts local models, while\nthe global model is preserved. As a result, and contrary to previous works, the\ngradients on local servers are computed on perturbed models. Consequently,\nconvergence proofs are more challenging and require a precise control of this\nperturbation. To ensure it, MCM additionally combines model compression with a\nmemory mechanism. This analysis opens new doors, e.g. incorporating worker\ndependent randomized-models and partial participation.\n