Optimal Communication-Computation Trade-Off in Heterogeneous Gradient Coding

Gradient coding allows a master node to derive the aggregate of the partial\ngradients, calculated by some worker nodes over the local data sets, with\nminimum communication cost, and in the presence of stragglers. In this paper,\nfor gradient coding with linear encoding, we characterize the optimum\ncommunication cost for heterogeneous distributed systems with \\emph{arbitrary}\ndata placement, with $s \\in \\mathbb{N}$ stragglers and $a \\in \\mathbb{N}$\nadversarial nodes. In particular, we show that the optimum communication cost,\nnormalized by the size of the gradient vectors, is equal to $(r-s-2a)^{-1}$,\nwhere $r \\in \\mathbb{N}$ is the minimum number that a data partition is\nreplicated. In other words, the communication cost is determined by the data\npartition with the minimum replication, irrespective of the structure of the\nplacement. The proposed achievable scheme also allows us to target the\ncomputation of a polynomial function of the aggregated gradient matrix. It also\nallows us to borrow some ideas from approximation computing and propose an\napproximate gradient coding scheme for the cases when the repetition in data\nplacement is smaller than what is needed to meet the restriction imposed on\ncommunication cost or when the number of stragglers appears to be more than the\npresumed value in the system design.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC