Corpus ID: 49868726

Beyond Data and Model Parallelism for Deep Neural Networks

@article{Jia2018BeyondDA,
  title={Beyond Data and Model Parallelism for Deep Neural Networks},
  author={Zhihao Jia and Matei Zaharia and Alexander Aiken},
  journal={ArXiv},
  year={2018},
  volume={abs/1807.05358}
}
  • Zhihao Jia, Matei Zaharia, Alexander Aiken
  • Published in ArXiv 2018
  • Computer Science
  • The computational requirements for training deep neural networks (DNNs) have grown to the point that it is now standard practice to parallelize training. [...] Key Method We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine.Expand Abstract

    Create an AI-powered research feed to stay up to date with new papers like this posted to ArXiv

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 42 CITATIONS

    Parallel Training via Computation Graph Transformation

    VIEW 18 EXCERPTS
    CITES METHODS & BACKGROUND
    HIGHLY INFLUENCED

    PipeDream: generalized pipeline parallelism for DNN training

    VIEW 6 EXCERPTS
    CITES BACKGROUND

    Proposal Machine Learning Parallelism Could Be Adaptive , Composable and Automated

    VIEW 11 EXCERPTS
    CITES BACKGROUND & METHODS
    HIGHLY INFLUENCED

    Reinforced Genetic Algorithm Learning for Optimizing Computation Graphs

    VIEW 3 EXCERPTS
    CITES METHODS
    HIGHLY INFLUENCED

    Simulating Performance of ML Systems with Offline Profiling

    VIEW 3 EXCERPTS
    CITES METHODS & BACKGROUND

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 40 REFERENCES

    Accurate

    • P. Goyal, P. Dollár, +6 authors K. He
    • large minibatch SGD: training imagenet in 1 hour. CoRR, abs/1706.02677
    • 2017
    VIEW 16 EXCERPTS
    HIGHLY INFLUENTIAL

    14 X

    • Y. Wu, M. Schuster, +10 authors M. Johnson
    • Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/160
    • 2016
    VIEW 21 EXCERPTS
    HIGHLY INFLUENTIAL

    Device Placement Optimization with Reinforcement Learning

    VIEW 10 EXCERPTS
    HIGHLY INFLUENTIAL

    Convolutional neural networks for sentence classification

    • Y. Kim
    • CoRR, abs/1408.5882
    • 2014
    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    One weird trick for parallelizing convolutional neural networks

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    Recurrent Neural Network Regularization

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    Deep Residual Learning for Image Recognition

    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    Modular

    • A New Lightweight
    • and Scalable Deep Learning Framework. https://caffe2.ai
    • 2016
    VIEW 10 EXCERPTS
    HIGHLY INFLUENTIAL