DeepPCR: Parallelizing Sequential Operations in Neural Networks

Parallelization techniques have become ubiquitous for accelerating inference and training of deep neural networks. Despite this, several operations are still performed in a sequential manner. For instance, the forward and backward passes are executed layer-by-layer, and the output of diffusion models is produced by applying a sequence of denoising steps. This sequential approach results in a computational cost proportional to the number of steps involved, presenting a potential bottleneck as the number of steps increases. In this work, we introduce DeepPCR, a novel algorithm which parallelizes typically sequential operations in order to speed up inference and training of neural networks. DeepPCR is based on interpreting a sequence of $L$ steps as the solution of a specific system of equations, which we recover using the Parallel Cyclic Reduction algorithm. This reduces the complexity of computing the sequential operations from $\mathcal{O}(L)$ to $\mathcal{O}(\log_2L)$, thus yielding a speedup for large $L$. To verify the theoretical lower complexity of the algorithm, and to identify regimes for speedup, we test the effectiveness of DeepPCR in parallelizing the forward and backward pass in multi-layer perceptrons, and reach speedups of up to $30\times$ for the forward and $200\times$ for the backward pass. We additionally showcase the flexibility of DeepPCR by parallelizing training of ResNets with as many as 1024 layers, and generation in diffusion models, enabling up to $7\times$ faster training and $11\times$ faster generation, respectively, when compared to the sequential approach.

Paper

References (44)

Scroll for more · 32 remaining

Similar papers

Peer review

Reviewer 7Bpu7/10 · confidence 3/52023-07-06

Summary

This work introduces DeepPCR, a method for parallelizing sequential computations with an application in neural networks. The method is based on the assumption that the target sequence is Markovian, and if so, DeepPCR can leverage Parallel Cyclic Reduction to convert the sequence into a parallel version that executes in O(log_2 L) instead of O(L). To account for the nonlinearity in neural networks, the authors approximate it using Newton’s method. They show that considerable wall-time speedup can be achieved using DeepPCR in both forward and backward passes on MLP/ResNet/diffusion process.

Strengths

* The authors tackle a significant problem using novel approaches. To solve the non-linearity, they propose to use Newton's method and prove that it requires few iterations when the non-linear operations (eg. activation) are smooth. * The evaluation section was convincing, and the limitations were properly discussed. * This work could have a profound impact on the parallelization techniques * The paper was overall well-written, with the main claims clearly articulated and properly supported by references and evaluations.

Weaknesses

* As one of the tradeoffs here is memory v.s. compute time, it would be great to see an analysis of space complexity with respect to the Jacobian blocks/model width, and a detailed memory profile breakdown at training time (which can be obtained using a profiler). * More evaluations on the transformer-based model will be welcome. The architecture of BERT/T5/GPT should satisfy the Markovian property, and I would expect DeepPCR to be able to work on them. * To evaluate the quality of stable diffusion models trained using DeepPCR, it would be great to present the FID and the generated images for quantitative and qualitative comparison.

Questions

* One important assumption made by DeepPCR is the Markov property. However, some layers/operators do require additional historical information or outputs of the previous layer. Could you clarify how to handle such cases, for instance, residual connections? Although you may fold the residual blocks(as discussed in the appendix), I wonder what will happen when the sizes of the residual blocks are non-uniform. Will the load of each parallel stage be unbalanced? * I am interested in the PCR overhead prior to training. How much time is required to perform PCR?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have properly addressed the limitations.

Reviewer Cpwv5/10 · confidence 4/52023-07-06

Summary

This paper presents a method called DeepPCR for converting sequential operations in Deep Neural Networks into parallel ones, thus accelerating DNN training, inference, and denoising procedure in diffusion models. The key idea of DeepPCR is to interpret a sequential operation of L steps that satisfies the Markov property as the solution of a system of L equations. DeepPCR tackles this solution by using the Parallel Cyclic Reduction algorithm (PCR) and Newton’s method in parallel, resulting in O(log L) steps compared to O(L) steps of the sequential operations. Evaluation results show DeepPCR brings up to 30X speedup for the forward pass and 200X speedup for the backward pass for certain cases in MLPs, and 11.2X speedup in image generation via diffusion. DeepPCR also accelerates ResNet training by up to 7X.

Strengths

+ This is a solid work that improves the parallel execution of DNNs from a more theoretical and fundamental perspective. + This paper is well written. Both the theory and system aspects of DeepPCR are carefully analyzed. + The resulting performance gains are significant.

Weaknesses

- The general idea of interpreting a sequential operation as the solution of a large system of equations is not brand new [36]. - It has a relatively strong assumption that the operation sequence needs to satisfy the Markov property, i.e., the output of each step depends only on that of the previous step and no past steps. - Some evaluation details seem missing or unclear.

Questions

In summary, this is an interesting work that aims to parallelize sequential operations in DNNs from a more theoretical and fundamental perspective. I generally enjoyed my reading and I have some detailed questions as follows besides the items in the weakness section above. It would be very helpful if the authors could provide more justifications: 1. DeepPCR currently works on sequential neural network structures that satisfy the Markov property. Is it a fundamental limitation? It would be helpful to explicitly list some neural architectures that follow (or do not follow) the Markov property to assist in understanding this limitation (if space allows). 2. DeepPCR focuses on execution performance. Some important evaluation specifications are necessary, e.g., GPU type and DNN training/inference framework/engine. 3. DeepPCR trains ResNet on the MNIST dataset. Why not use ImageNet that might be more convincing? 4. What is the relationship between Deep PCR’s design and the effort in operator fusion (and subsequent efficient parallelization)? It would be extremely helpful to include some discussion about this. 5. PCR requires extra memory (and results in more intensive computation) than its sequential counterpart, thus sensitive to the available hardware. How does it affect parallel efficiency? Does this cause any problems for large-scale training tasks that require distributed settings and intensive communications among nodes?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are carefully explained.

Reviewer uCYF5/10 · confidence 3/52023-07-07

Summary

In this paper, the authors proposed an algorithm (DeepPCR) which parallelizes typically sequential operations used in inference and training of neural networks. DeepPCR is based on interpreting a sequence of L steps as the solution of a specific system of equations, which got recovered using the Parallel Cyclic Reduction algorithm. To verify the the effectiveness of DeepPCR, authors presented the results on multi-layer perceptrons, and reach speedups of up to 30x for forward and 200x for backward pass. In addition, with ResNets with as many as 1024 layers, and generation in diffusion models, DeepPCR enables up to 7x faster training and 11x faster generation, respectively.

Strengths

1. In this paper, authors proposed to interpret a sequential operation of L steps as the solution of a system of L equations, which can be recovered using the Parallel Cyclic Reduction algorithm. 2. The paper is organized and written well. 3. Compared with previous work, instead of relying on variations of Jacobi iterations, authors proposed our method specifically targets Markov sequences. 4. In the experiment result section, authors have provided a comprehensive result to demonstrate the applicability of DeepPCR to a variety of scenarios.

Weaknesses

1. The motivation is not very clear. With the constraints of deepPCR, how much real-time saving is not shown in the result. That would be more helpful to understand if the speedup for neural networks sampling on dataset is provided. 2. Compared with Jacobi iterations

Questions

1. Which platform are the experiments in Sec. 4.1 performing at, CPU or GPU? In the appendix, deepPCR rerun for two GPUs: V100 and A100, but didn't see the result of baseline on these two GPUs. 2. Compared with Jacobi iterations in [36], is there any experimental comparison proving deepPCR have more benefit, such as ResNets?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

As mentioned in the limitation part, the proposed deepPCR has three main limitations: 1. PCR requires fewer sequential steps overall which needs deployments efficiently, otherwise lead to performance degradation. 2. The complexity actually becomes O(cN * log2 L), where cN identifies the number of Newton iterations necessary for convergence. There is a trade-off between complexity and accuracy. 3. Memory increment for the temporary results.

Reviewer 7Bpu2023-08-15

Thanks for the authors' detailed response!

Area Chair cfFv2023-08-18

Smoothness

Hi, thank you for your detailed responses. Could you please comment on the convergence of the Newton’s method? Generally speaking, it is a local method. It would be helpful to discuss its convergence leveraging the special structure of the problem. Under what assumptions does it converge? What tricks (e.g., initialization) are needed to make it converge in practice? I can imagine that if we have a sequence of L hash functions, the method will not work. So some smoothness would be necessary. It would be nice if the algorithm reduced to the sequential complexity O(L) (and still produced correct results) in these cases instead of not converging / producing wrong results. Best, AC

Authorsrebuttal2023-08-18

> Hi, thank you for your detailed responses. Could you please comment on the convergence of the Newton’s method? Generally speaking, it is a local method. It would be helpful to discuss its convergence leveraging the special structure of the problem. Under what assumptions does it converge? What tricks (e.g., initialization) are needed to make it converge in practice? I can imagine that if we have a sequence of L hash functions, the method will not work. So some smoothness would be necessary. We thank the AC for their question. Indeed, the performance of our method heavily depends on the behaviour of the Newton solver, and investigating its convergence property in this setting is paramount: this is the reason behind our experiments in Sec4.4 and SecE. In general, providing theoretical global convergence guarantees for Newton is not practical (if at all feasible!). Our target system (3) does present a very specific structure, though, which guarantees some useful properties, namely - The Jacobian is always invertible: notice it will always have identities on the main diagonal, hence all its eigenvalues are 1. This implies that the Newton sequence is well-defined - The root is unique and hence has multiplicity one. This stems from the uniqueness of the solution of the forward substitution (sequential) procedure Still, convergence will depend also on the (higher-order derivatives of the) step functions $f_l(z)$ which is ultimately heavily problem- and architecture-dependent. For this reason, we focused more on performing an experimental analysis rather than a theoretical one. In general, for the problems considered we didn’t encounter issues in terms of convergence: this can be seen in Fig7, where Newton iterations remain reasonably bounded even though a non-differentiable function (ReLU) was employed. This extends to the more complex case of diffusion, too (see SecD.3). However, the AC is absolutely correct in that, if we consider extreme cases of non-differentiability (such as argmax or hash functions), the Newton procedure is likely to fail (see also reply to W2 of reviewer 7Bpu). We will make sure to further highlight this in the limitations section. Regarding the role of initialisation, as the AC rightly assumes, we did observe that properly choosing an initial guess for the Newton solver plays a relevant role in reducing the number of iterations to convergence. Nonetheless, for the experiments considered, even employing relatively simple heuristics resulted in reasonably fast convergence: this is investigated in detail in SecE.2. For example, as an initial guess for Newton applied to the training procedure in ResNets we resorted to using the batch-average of the activations at the previous optimisation step, but already with an all-zero initialisation we can observe convergence, albeit slower (see in particular Fig18 for the effect of varying this). > It would be nice if the algorithm reduced to the sequential complexity O(L) (and still produced correct results) in these cases instead of not converging / producing wrong results. Best, AC We thoroughly agree: this would make for an ideal property (and indeed the Jacobi solver in [36] does satisfy this), but it is just not feasible for our choice of solver. This is an expected trade-off for a (generally) faster solver (Jacobi is linear, Newton is quadratic). The classical workaround consists in switching solving strategy depending on the behaviour of the first (few) iterations: if Newton diverges, one can always revert to the sequential solution (although the cost of the first few iterations is indeed wasted in this case). We will explicitly point this out, too, in the paper. We thank again the AC for their questions, and remain at their disposal for any additional clarification.

Reviewer Cpwv2023-08-21

Thanks for the authors' careful response!

Authorsrebuttal2023-08-21

Thanks again to you for the valuable feedback you’ve provided in your review! If you feel like we’ve addressed your concerns in our rebuttal, we would be grateful if you considered increasing the score provided. If not, and some doubts still remain, we would appreciate if you could further point them out, so that we can try and address them properly.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC