Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms

Communication-efficient SGD algorithms, which allow nodes to perform local\nupdates and periodically synchronize local models, are highly effective in\nimproving the speed and scalability of distributed SGD. However, a rigorous\nconvergence analysis and comparative study of different communication-reduction\nstrategies remains a largely open problem. This paper presents a unified\nframework called Cooperative SGD that subsumes existing communication-efficient\nSGD algorithms such as periodic-averaging, elastic-averaging and decentralized\nSGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for\nexisting algorithms. Moreover, this framework enables us to design new\ncommunication-efficient SGD algorithms that strike the best balance between\nreducing communication overhead and achieving fast error convergence with low\nerror floor.\n