Deep learning in neural networks: An overview

@article{Schmidhuber2015DeepLI,
  title={Deep learning in neural networks: An overview},
  author={J. Schmidhuber},
  journal={Neural networks : the official journal of the International Neural Network Society},
  year={2015},
  volume={61},
  pages={
          85-117
        }
}
  • J. Schmidhuber
  • Published 2015
  • Computer Science, Medicine
  • Neural networks : the official journal of the International Neural Network Society
In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of… Expand
Chapter 7 – Unsupervised deep learning: A short review
TLDR
This chapter considers problems with training deep networks using backpropagation-type algorithms, and considers various structures used in deep learning, including restricted Boltzmann machines, deep belief networks, deep Boltzman machines, and nonlinear autoencoders. Expand
Deep Reinforcement Learning: An Overview
TLDR
This article reviews the recent advances in deep reinforcement learning with focus on the most used deep architectures such as autoencoders, convolutional neural networks and recurrent neural networks which have successfully been come together with the reinforcement learning framework. Expand
Deep Learning: Architectures and Applications
TLDR
This chapter covers the basics ofDeep learning, different architectures of deep learning like artificial neural network, feed forward neural network), CNN, CNN, recurrent neuralnetwork, deep Boltzmann machine, and their comparison and summarizes the applications ofdeep learning in different areas. Expand
A Shallow Introduction to Deep Neural Networks
TLDR
This chapter familiarizes the readers with the major classes of deep neural networks that are frequently used, namely CNN (Convolutional neural Network), RNN (Recurrent Neural Network), DBN (Deep Belief Network), Deep autoencoder, GAN (Generative Adversarial Network) and Deep Recursive Network. Expand
A Comprehensive Study of Deep Neural Networks for Unsupervised Deep Learning
TLDR
This chapter first study difficulties with neural networks while training with backpropagation-algorithms, then different structures, namely, restricted Boltzmann machines (RBMs), Deep Belief Networks (DBNs), nonlinear autoencoders, deep BoltZmann machines are covered. Expand
A Tutorial on Deep Neural Networks for Intelligent Systems
TLDR
A tutorial of Deep Neural Networks (DNNs), and some insights about the origin of the term "deep"; references to deep learning are given. Expand
An Overview Studying of Deep Learning
Deep learning is a subfield of machine learning however both drop under the broad category of artificial intelligence. Deep learning is what powers the most human-like artificial intelligence thatExpand
Deep learning and Its Development
Deep learning is a promising branch of machine learning. It is an algorithm that uses artificial neural networks as the architecture to characterize and learn data. In recent years, many companies,Expand
Recent Advances in Deep Learning: An Overview
TLDR
This paper is going to briefly discuss about recent advances in Deep Learning for past few years. Expand
Neural network models and deep learning
TLDR
A brief introduction to neural network models and deep learning for biologists is given and the expressive power of this modeling framework and the backpropagation algorithm for setting the parameters are explained. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 1,183 REFERENCES
Unsupervised Learning in LSTM Recurrent Neural Networks
TLDR
Long Short-Term Memory recurrent networks are trained to maximize two information-theoretic objectives for unsupervised learning: Binary Information Gain Optimization (BINGO) and Nonparametric Entropyoptimization (NEO). Expand
Co-evolving recurrent neurons learn deep memory POMDPs
TLDR
A new neuroevolution algorithm called Hierarchical Enforced SubPopulations that simultaneously evolves networks at two levels of granularity: full networks and network components or neurons is introduced. Expand
Neural Networks: Tricks of the Trade
TLDR
It is shown how nonlinear semi-supervised embedding algorithms popular for use with â œshallowâ learning techniques such as kernel methods can be easily applied to deep multi-layer architectures. Expand
Adaptive dropout for training deep neural networks
TLDR
A method is described called 'standout' in which a binary belief network is overlaid on a neural network and is used to regularize of its hidden units by selectively setting activities to zero, which achieves lower classification error rates than other feature learning methods, including standard dropout, denoising auto-encoders, and restricted Boltzmann machines. Expand
Unsupervised Learning Procedures for Neural Networks
  • S. Becker
  • Computer Science
  • Int. J. Neural Syst.
  • 1991
TLDR
This paper describes the major approaches that have been taken to model unsupervised learning, and gives an in-depth review of several examples of each approach. Expand
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporalExpand
How to Construct Deep Recurrent Neural Networks
TLDR
Two novel architectures of a deep RNN are proposed which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build aDeep RNN, and an alternative interpretation is provided using a novel framework based on neural operators. Expand
Learning long-term dependencies with gradient descent is difficult
TLDR
This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods. Expand
Forecasting with Recurrent Neural Networks: 12 Tricks
TLDR
This paper starts from the fact that RNNs can model any high dimensional, nonlinear dynamical system, and focuses on the design of network architectures. Expand
Greedy Layer-Wise Training of Deep Networks
TLDR
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization. Expand
...
1
2
3
4
5
...