Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid

A Multigrid Full Approximation Storage algorithm for solving Deep Residual\nNetworks is developed to enable neural network parallelized layer-wise training\nand concurrent computational kernel execution on GPUs. This work demonstrates a\n10.2x speedup over traditional layer-wise model parallelism techniques using\nthe same number of compute units.\n

cs.LG cs.DC cs.PF stat.ML Advanced Neural Network Applications Medical Imaging and Analysis Parallel Computing and Optimization Techniques Computer Science Engineering Mathematics

View graph Source

Paper

Similar papers

Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear MultigridAndrew C. Kirby, Siddharth Samsi, Michael Jones et al.2020 In Library

Layer-Parallel Training of Deep Residual Neural NetworksStefanie Günther, Lars Ruthotto, Jacob B. Schroder et al.2020 · SIAM Journal on Mathematics of Data Science In Library

Layer-Parallel Training of Deep Residual Neural NetworksStefanie Günther, Lars Ruthotto, J. Schroder et al.2018 · SIAM Journal on Mathematics of Data Science · 107 citations In Library

Parallelizing Over Artificial Neural Network Training Runs with MultigridJ. Schroder2017 · arXiv.org · 7 citations In Library

Exploring Hidden Dimensions in Parallelizing Convolutional Neural NetworksZhihao Jia, Sina Lin, C. Qi et al.2018 · International Conference on Machine Learning · 131 citations In Library

Multilevel Initialization for Layer-Parallel Deep Neural Network TrainingE. Cyr, Stefanie Günther, J. Schroder2019 · arXiv.org · 12 citations In Library

Parareal Neural Networks Emulating a Parallel-in-time AlgorithmZhanyu Ma, Yuping Lai, Jiyang Xie et al.2021 · IEEE Transactions on Neural Networks and Learning Systems · 13 citations In Library

A Practical Layer-Parallel Training Algorithm for Residual NetworksQi Sun, Hexin Dong, Zewei Chen et al.2020 · arXiv (Cornell University) In Library

Parallel Training of GRU Networks with a Multi-Grid Solver for Long SequencesGordon Euhyun Moon, Eric C. Cyr2022 · arXiv (Cornell University) · 9 citations In Library

Layer-Parallel Training of Residual Networks with Auxiliary-Variable NetworksQi Sun, Hexin Dong, Zewei Chen et al.2021 · arXiv (Cornell University) In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal