Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications

This paper characterizes the interplay between overlapping compute and communication in GPU-accelerated distributed training, a critical but often overlooked aspect impacting performance and power. We systematically evaluate state-of-the-art NVIDIA (H100/A100) and AMD (MI250/MI210) GPUs, investigating the effects of hardware features including numeric precision, specialized cores, and power capping. Experiments across GPT and LLaMA models show that overlap causes significant compute slowdown (an average of 18.9 % and upto 40.0 %) compared to an ideal scenario where compute executes without communication interference. However, overlapping execution is still faster than sequential execution by an average 10.2 % and maximum of 26.6 %. We find overlap increases peak power, specialized datapaths have mixed effects, and power capping severely exacerbates slowdowns. These results emphasize the need for balanced strategies in optimizing distributed training throughput and power efficiency.