# A block-random algorithm for learning on distributed, heterogeneous data

@article{Mohan2019ABA, title={A block-random algorithm for learning on distributed, heterogeneous data}, author={Prakash Mohan and Marc T. Henry de Frahan and Ryan N. King and Ray W. Grout}, journal={ArXiv}, year={2019}, volume={abs/1903.00091} }

Most deep learning models are based on deep neural networks with multiple layers between input and output. The parameters defining these layers are initialized using random values and are "learned" from data, typically using stochastic gradient descent based algorithms. These algorithms rely on data being randomly shuffled before optimization. The randomization of the data prior to processing in batches that is formally required for stochastic gradient descent algorithm to effectively derive a… Expand

#### Supplemental Code

#### References

SHOWING 1-10 OF 33 REFERENCES

Parallelized Stochastic Gradient Descent

- Computer Science, Mathematics
- NIPS
- 2010

This paper presents the first parallel stochastic gradient descent algorithm including a detailed analysis and experimental evidence and introduces a novel proof technique — contractive mappings to quantify the speed of convergence of parameter distributions to their asymptotic limits. Expand

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2011

This work describes and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal functions that can be chosen in hindsight. Expand

Large-Scale Machine Learning with Stochastic Gradient Descent

- Computer Science
- COMPSTAT
- 2010

A more precise analysis uncovers qualitatively different tradeoffs for the case of small-scale and large-scale learning problems. Expand

Adam: A Method for Stochastic Optimization

- Computer Science, Mathematics
- ICLR
- 2015

This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Expand

Deep learning in neural networks: An overview

- Computer Science, Medicine
- Neural Networks
- 2015

This historical survey compactly summarizes relevant work, much of it from the previous millennium, review deep supervised learning, unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. Expand

A neural network approach for the blind deconvolution of turbulent flows

- Physics, Computer Science
- Journal of Fluid Mechanics
- 2017

The proposed blind deconvolution network performs exceptionally well in the a priori testing of two-dimensional Kraichnan, three-dimensional Kolmogorov and compressible stratified turbulence test cases, and shows promise in forming the backbone of a physics-augmented data-driven closure for the Navier–Stokes equations. Expand

Neural networks: An overview of early research, current frameworks and new challenges

- Computer Science
- Neurocomputing
- 2016

The development and evolution of different topics related to neural networks is described showing that the field has acquired maturity and consolidation, proven by its competitiveness in solving real-world problems. Expand

EMNIST: an extension of MNIST to handwritten letters

- Computer Science
- CVPR 2017
- 2017

A variant of the full NIST dataset is introduced, which is called Extended MNIST (EMNIST), which follows the same conversion paradigm used to create the MNIST dataset, and shares the same image structure and parameters as the original MNIST task, allowing for direct compatibility with all existing classifiers and systems. Expand

New Approaches in Turbulence and Transition Modeling Using Data-driven Techniques

- Computer Science
- 2015

A data-driven approach to the modeling of turbulent and transitional flows is proposed, to infer the functional form of deficiencies in known closure models by applying inverse problems to computational and experimental data, use machine learning to reconstruct the improved functional forms, and to inject the improvedfunctional forms in simulations to obtain more accurate predictions. Expand

Scikit-learn: Machine Learning in Python

- Computer Science
- J. Mach. Learn. Res.
- 2011

Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing… Expand