Deep learning in neural networks: An overview

  title={Deep learning in neural networks: An overview},
  author={J{\"u}rgen Schmidhuber},
  journal={Neural networks : the official journal of the International Neural Network Society},
  • J. Schmidhuber
  • Published 30 April 2014
  • Computer Science
  • Neural networks : the official journal of the International Neural Network Society

Deep Reinforcement Learning: An Overview

This article reviews the recent advances in deep reinforcement learning with focus on the most used deep architectures such as autoencoders, convolutional neural networks and recurrent neural networks which have successfully been come together with the reinforcement learning framework.

A Shallow Introduction to Deep Neural Networks

This chapter familiarizes the readers with the major classes of deep neural networks that are frequently used, namely CNN (Convolutional neural Network), RNN (Recurrent Neural Network), DBN (Deep Belief Network), Deep autoencoder, GAN (Generative Adversarial Network) and Deep Recursive Network.

Deep Learning

This chapter covers the basics ofDeep learning, different architectures of deep learning like artificial neural network, feed forward neural network), CNN, CNN, recurrent neuralnetwork, deep Boltzmann machine, and their comparison and summarizes the applications ofdeep learning in different areas.

A Comprehensive Study of Deep Neural Networks for Unsupervised Deep Learning

This chapter first study difficulties with neural networks while training with backpropagation-algorithms, then different structures, namely, restricted Boltzmann machines (RBMs), Deep Belief Networks (DBNs), nonlinear autoencoders, deep BoltZmann machines are covered.

An Overview Studying of Deep Learning

The main advantage of Deep Learning is to create an artificial neural network that can learn and make intelligent decisions on its own and to process large numbers of features makes deep learning very powerful when dealing with unstructured data.

A Tutorial on Deep Neural Networks for Intelligent Systems

A tutorial of Deep Neural Networks (DNNs), and some insights about the origin of the term "deep"; references to deep learning are given.

Deep learning and Its Development

The current application of deep learning in various fields is summarized, such as artificial intelligence, computer vision and natural language processing applications, and some open problems for future research are also analyzed.

Recent Advances in Deep Learning: An Overview

This paper is going to briefly discuss about recent advances in Deep Learning for past few years.

Neural network models and deep learning




Unsupervised Learning in LSTM Recurrent Neural Networks

Long Short-Term Memory recurrent networks are trained to maximize two information-theoretic objectives for unsupervised learning: Binary Information Gain Optimization (BINGO) and Nonparametric Entropyoptimization (NEO).

Neural Networks: Tricks of the Trade

It is shown how nonlinear semi-supervised embedding algorithms popular for use with â œshallowâ learning techniques such as kernel methods can be easily applied to deep multi-layer architectures.

Co-evolving recurrent neurons learn deep memory POMDPs

A new neuroevolution algorithm called Hierarchical Enforced SubPopulations that simultaneously evolves networks at two levels of granularity: full networks and network components or neurons is introduced.

Adaptive dropout for training deep neural networks

A method is described called 'standout' in which a binary belief network is overlaid on a neural network and is used to regularize of its hidden units by selectively setting activities to zero, which achieves lower classification error rates than other feature learning methods, including standard dropout, denoising auto-encoders, and restricted Boltzmann machines.

Unsupervised Learning Procedures for Neural Networks

  • S. Becker
  • Computer Science
    Int. J. Neural Syst.
  • 1991
This paper describes the major approaches that have been taken to model unsupervised learning, and gives an in-depth review of several examples of each approach.

A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal

Learning long-term dependencies with gradient descent is difficult

This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.

How to Construct Deep Recurrent Neural Networks

Two novel architectures of a deep RNN are proposed which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build aDeep RNN, and an alternative interpretation is provided using a novel framework based on neural operators.

Greedy Layer-Wise Training of Deep Networks

These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.

Learning Deep Architectures for AI

The motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer modelssuch as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks are discussed.