Model-Aware Parallelization Strategy for Deep Neural Networks' Distributed Training

Abstract

It is standard practice today to parallelize deep neural networks' (DNNs') training across distributed clusters. The most common parallelization strategy is data parallelism, but its speedup is limited by tremendous communication overhead. To alleviate this problem, we employ hybrid parallelism, where the size of data to be transferred is decreased. Hybrid parallelism divides machines into groups. Machines in the same group maintain different parts of the model and carry out training tasks collaboratively, and all the groups perform computations simultaneously. Based on hybrid parallelism, we propose a Model-aware Parallelization Strategy, with the purpose of completing the training task as quickly as possible. To determine which strategy to use, we formulate a combinatorial optimization problem and use a greedy heuristic algorithm to decide how to group machines and how to partition the network. We evaluate our Model-aware Parallelization Strategy with six frequently-used deep neural networks. Experimental results show that, for large networks such as VGG, the Model-aware Parallelization Strategy reduces completion time by more than 20 percent.