LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks

In our prior work, LayerPipe, we had introduced an approach to accelerate training of convolutional, fully connected, and spiking neural networks by overlapping forward and backward computation. However, despite empirical success, a principled understanding of how much gradient delay needs to be introduced at each layer to achieve desired level of pipelining was not addressed. This paper, LayerPipe2, fills that gap by formally deriving LayerPipe using variable delayed–gradient adaptation and retiming. We identify where delays may be legally inserted and show that the required amount of delay follows directly from the network structure: inner layers require fewer delays, while outer layers require longer delays. When pipelining is applied at every layer, each delay depends only on the number of remaining downstream stages; when layers are pipelined in groups, all layers in the group share the same assignment. These insights not only explain previously observed scheduling patterns but also expose an often-overlooked challenge: pipelining implicitly requires storage of historical weights. We overcome this storage bottleneck by developing a pipeline–aware moving average that reconstructs the required past states rather than storing them explicitly. This reduces memory cost without sacrificing the accuracy guarantees that makes pipelined learning viable. The result is a principled framework that illustrates how to construct LayerPipe architectures, predicts their delay requirements, and mitigates their storage burden, thereby enabling scalable pipelined training with controlled communication–computation tradeoffs.