One weird trick for parallelizing convolutional neural networks

I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.

cs.NE cs.DC cs.LG Computer Science

Source

Paper

References (8)

01GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network TrainingT. Paine, Hailin Jin, Jianchao Yang et al.2013 · International Conference on Learning Representations · 100 citations In Library

02Multi-GPU Training of ConvNetsOmry Yadan, Keith Adams, Yaniv Taigman et al.2013 · International Conference on Learning Representations · 105 citations In Library

03Deep learning with COTS HPC systemsA. Coates, Brody Huval, Tao Wang et al.2013 · International Conference on Machine Learning · 755 citations

04ImageNet classification with deep convolutional neural networksA. Krizhevsky, I. Sutskever, Geoffrey E. Hinton2012 · Communications of the ACM · 131k citations

05Large Scale Distributed Deep NetworksJ. Dean, G. Corrado, R. Monga et al.2012 · Neural Information Processing Systems · 4.2k citations

06Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient DescentB. Recht, Christopher Ré, Stephen J. Wright et al.2011 · Neural Information Processing Systems · 2.4k citations In Library

07ImageNet: A large-scale hierarchical image databaseJia Deng, Wei Dong, R. Socher et al.2009 · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 75k citations

08One of the workers sends its last-stage convolutional layer activities to all other workersparallel with this computation

Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUsBehnam Pourghassemi, Chenghao Zhang, Joo Hwan Lee et al.2020 · arXiv (Cornell University) · 12 citations In Library

On the Limits of Parallelizing Convolutional Neural Networks on GPUsBehnam Pourghassemi, Chenghao Zhang, Joo Hwan Lee et al.2020 In Library

Regularized Convolutional Neural Network for Highly Effective Parallel ProcessingSangsoo Park, Ki‐Seok Chung2022 · Journal of Computing Science and Engineering · 1 citation In Library

Distributed Learning of CNNs on Heterogeneous CPU/GPU ArchitecturesJose Marques, G. F. P. Fernandes, Luís A. Alexandre2017 · Applied Artificial Intelligence · 5 citations In Library

Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous ArchitecturesFelipe Farias, Teresa B. Ludermir, Carmelo J. A. Bastos-Filho2022 · AI In Library

Optimizing Performance of Recurrent Neural Networks on GPUsJ. Appleyard, Tomás Kociský, Phil Blunsom2016 · arXiv.org · 95 citations In Library

Accelerating Convolutional Neural Network by Exploiting Sparsity on GPUsWeizhi Xu, Yintai Sun, Shengyu Fan et al.2023 · ACM Transactions on Architecture and Code Optimization In Library

Accelerating Convolutional Neural Network by Exploiting Sparsity on GPUsWeizhi Xu, Yintai Sun, Shengyu Fan et al.2019 · ACM Transactions on Architecture and Code Optimization (TACO) · 12 citations In Library

Parareal Neural Networks Emulating a Parallel-in-time AlgorithmZhanyu Ma, Yuping Lai, Jiyang Xie et al.2021 · IEEE Transactions on Neural Networks and Learning Systems · 13 citations In Library

Patents near this paper

EFFICIENT PARALLEL TRAINING OF A NETWORK MODEL ON MULTIPLE GRAPHICS PROCESSING UNITSUS 10,949,746 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2021 SYSTEM AND METHOD FOR PARALLELIZING CONVOLUTIONAL NEURAL NETWORKSUS 10,635,966 · DNNRESEARCH INC. · 2020 SYSTEM AND METHOD FOR PARALLELIZING CONVOLUTIONAL NEURAL NETWORKSUS 9,563,840 · DNNRESEARCH INC. · 2017 PARALLELIZING NEURAL NETWORKS DURING TRAININGUS 9,811,775 · DNNRESEARCH INC. · 2017 High Performance Portable Convulational Neural Network Library on GP-GPUsUS 9,904,976 · NEC LABS AMERICA · 2018

Source of record

arXiv

One weird trick for parallelizing convolutional neural networks

Paper

References (8)

Similar papers

Patents near this paper

Welcome.

Patents near this paper