Scaling deep learning on GPU and knights landing clusters

@article{You2017ScalingDL,
  title={Scaling deep learning on GPU and knights landing clusters},
  author={Yang You and Aydin Buluç and J. Demmel},
  journal={Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
  year={2017}
}
  • Yang You, Aydin Buluç, J. Demmel
  • Published 2017
  • Computer Science
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
  • Training neural networks has become a big bottleneck. For example, training ImageNet dataset on one Nvidia K20 GPU needs 21 days. To speed up the training process, the current deep learning systems heavily rely on the hardware accelerators. However, these accelerators have limited on-chip memory compared with CPUs. We use both self-host Intel Knights Landing (KNL) clusters and multi-GPU clusters as our target platforms. From the algorithm aspect, we focus on Elastic Averaging SGD (EASGD) to… CONTINUE READING
    44 Citations
    On Linear Learning with Manycore Processors
    • 1
    • PDF
    Evaluation of On-Node GPU Interconnects for Training Deep Neural Networks
    • 1
    • PDF
    A Quantitative Study of Deep Learning Training on Heterogeneous Supercomputers
    • 4
    • Highly Influenced
    • PDF
    A survey of techniques for optimizing deep learning on GPUs
    • 19
    Hierarchical Distributed-Memory Multi-Leader MPI-Allreduce for Deep Learning Workloads
    • 5
    Reducing Data Motion to Accelerate the Training of Deep Neural Networks
    Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs
    • 5
    • Highly Influenced
    An Overview of Efficient Interconnection Networks for Deep Neural Network Accelerators
    MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms
    • 21
    • PDF

    References

    SHOWING 1-8 OF 8 REFERENCES
    Deep learning with COTS HPC systems
    • 602
    • Highly Influential
    • PDF
    On the importance of initialization and momentum in deep learning
    • 2,593
    • Highly Influential
    • PDF
    Deep learning with Elastic Averaging SGD
    • 370
    • Highly Influential
    • PDF
    Going deeper with convolutions
    • 20,955
    • Highly Influential
    • PDF
    Deep Learning
    • 14,261
    • Highly Influential
    • PDF
    GradientBased Learning Applied to Document Recognition
    • 2,636
    • Highly Influential
    Gradient-based learning applied to document recognition
    • 25,594
    • Highly Influential
    • PDF
    Imagenet classi€cation with deep convolutional neural networks. In Advances in neural information processing
    • 2012