Corpus ID: 67856745

A block-random algorithm for learning on distributed, heterogeneous data

  title={A block-random algorithm for learning on distributed, heterogeneous data},
  author={Prakash Mohan and Marc T. Henry de Frahan and Ryan N. King and Ray W. Grout},
Most deep learning models are based on deep neural networks with multiple layers between input and output. The parameters defining these layers are initialized using random values and are "learned" from data, typically using stochastic gradient descent based algorithms. These algorithms rely on data being randomly shuffled before optimization. The randomization of the data prior to processing in batches that is formally required for stochastic gradient descent algorithm to effectively derive a… Expand


Parallelized Stochastic Gradient Descent
This paper presents the first parallel stochastic gradient descent algorithm including a detailed analysis and experimental evidence and introduces a novel proof technique — contractive mappings to quantify the speed of convergence of parameter distributions to their asymptotic limits. Expand
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
This work describes and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal functions that can be chosen in hindsight. Expand
Large-Scale Machine Learning with Stochastic Gradient Descent
A more precise analysis uncovers qualitatively different tradeoffs for the case of small-scale and large-scale learning problems. Expand
Adam: A Method for Stochastic Optimization
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Expand
Deep learning in neural networks: An overview
This historical survey compactly summarizes relevant work, much of it from the previous millennium, review deep supervised learning, unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. Expand
A neural network approach for the blind deconvolution of turbulent flows
The proposed blind deconvolution network performs exceptionally well in the a priori testing of two-dimensional Kraichnan, three-dimensional Kolmogorov and compressible stratified turbulence test cases, and shows promise in forming the backbone of a physics-augmented data-driven closure for the Navier–Stokes equations. Expand
Neural networks: An overview of early research, current frameworks and new challenges
The development and evolution of different topics related to neural networks is described showing that the field has acquired maturity and consolidation, proven by its competitiveness in solving real-world problems. Expand
EMNIST: an extension of MNIST to handwritten letters
A variant of the full NIST dataset is introduced, which is called Extended MNIST (EMNIST), which follows the same conversion paradigm used to create the MNIST dataset, and shares the same image structure and parameters as the original MNIST task, allowing for direct compatibility with all existing classifiers and systems. Expand
New Approaches in Turbulence and Transition Modeling Using Data-driven Techniques
A data-driven approach to the modeling of turbulent and transitional flows is proposed, to infer the functional form of deficiencies in known closure models by applying inverse problems to computational and experimental data, use machine learning to reconstruct the improved functional forms, and to inject the improvedfunctional forms in simulations to obtain more accurate predictions. Expand
Scikit-learn: Machine Learning in Python
Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringingExpand