Corpus ID: 222208763

Interlocking Backpropagation: Improving depthwise model-parallelism

@article{Gomez2020InterlockingBI,
  title={Interlocking Backpropagation: Improving depthwise model-parallelism},
  author={Aidan N. Gomez and Oscar Key and Stephen Gou and Nick Frosst and J. Dean and Y. Gal},
  journal={ArXiv},
  year={2020},
  volume={abs/2010.04116}
}

• Aidan N. Gomez, Oscar Key, +3 authors Y. Gal
• Published 2020
• Computer Science
• ArXiv

The number of parameters in state of the art neural networks has drastically increased in recent years. This surge of interest in large scale neural networks has motivated the development of new distributed training strategies enabling such models. One such strategy is model-parallel distributed training. Unfortunately, model-parallelism suffers from poor resource utilisation, which leads to wasted resources. In this work, we improve upon recent developments in an idealised model-parallel… Expand

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed