Learning multiple layers of representation

  title={Learning multiple layers of representation},
  author={Geoffrey E. Hinton},
  journal={Trends in Cognitive Sciences},

Figures from this paper

Autonomous Learning of Representations
The goal of this contribution is to give an overview about different principles of autonomous feature learning, and to exemplify two principles based on two recent examples: autonomous metric learning for sequences, and autonomous learning of a deep representation for spoken language, respectively.
Learning Representations from Deep Networks Using Mode Synthesizers
This paper seeks to discover and represent interesting patterns in an efficient and less complex way by incorporating the concept of Mode synthesizers in the deep learning process models.
Modeling language and cognition with deep unsupervised learning: a tutorial overview
It is argued that the focus on deep architectures and generative (rather than discriminative) learning represents a crucial step forward for the connectionist modeling enterprise, because it offers a more plausible model of cortical learning as well as a way to bridge the gap between emergentist connectionist models and structured Bayesian models of cognition.
Contributions to Deep Learning Models
Two novel methods proposed: a new regularization technique for Restricted Boltzmann Machines called Mask Selective Regularization (MSR), and a powerful discriminative network called Local Deep Neural Network (Local-DNN) which improves other state-of-the-art results.
Learning Deep Visual Representations
This thesis represents one of the first attempts to synthesize deep learning and the bag-of-words model and introduces the notion of gradually transiting from unsupervised layer-wise learning to supervised deep learning.
Deep learning models of biological visual information processing
The proposed deep models outperform DBNs and DNNs on face completion and dimensionality reduction, thereby demonstrating the strength of methods inspired by biological visual processing.
A review on advances in deep learning
  • Soniya, Sandeep Paul, Lotika Singh
  • Computer Science
    2015 IEEE Workshop on Computational Intelligence: Theories, Applications and Future Directions (WCI)
  • 2015
The paper presents an overview of different deep learning approaches in a nutshell and also highlights some limitations which are restricting performance of deep neural networks in order to handle more realistic problems.
Vector LIDA
Deep Learning in Kernel Machines
Three deep kernel learning models are developed that analyze the behavior of arc-cosine kernel and modeled a scalable deep kernel machine by incorporating arc- cosine kernel in core vector machines, and a scalableDeep learning architecture with unsupervised feature extraction with promising results.
Interpreting faces with neurally inspired generative models
The studies showed that neural networks that extract highly nonlinear features from images using architectures inspired by the brain can solve difficult face perception tasks with minimal guidance by human experts.


A multi-layer sparse coding network learns contour coding from natural images
Learning representations by back-propagating errors
Back-propagation repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector, which helps to represent important features of the task domain.
Greedy Layer-Wise Training of Deep Networks
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
Reducing the Dimensionality of Data with Neural Networks
This work describes an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.
Scaling learning algorithms towards AI
It is argued that deep architectures have the potential to generalize in non-local ways, i.e., beyond immediate neighbors, and that this is crucial in order to make progress on the kind of complex tasks required for artificial intelligence.
Learning Multilevel Distributed Representations for High-Dimensional Sequences
A new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems are described, and their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.
Modeling image patches with a directed hierarchy of Markov random fields
An efficient learning procedure for multilayer generative models that combine the best aspects of Markov random fields and deep, directed belief nets is described and it is shown that this type of model is good at capturing the statistics of patches of natural images.
A Fast Learning Algorithm for Deep Belief Nets
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
Unsupervised Learning of Image Transformations
A probabilistic model for learning rich, distributed representations of image transformations that develops domain specific motion features, in the form of fields of locally transformed edge filters, and can fantasize new transformations on previously unseen images.
Learning higher-order structures in natural images
A hierarchical probabilistic model for learning higher-order statistical regularities in natural images is presented and could provide theoretical insight into the response properties and computational functions of lower level cortical visual areas.