Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

  title={Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations},
  author={Honglak Lee and Roger Baker Grosse and Rajesh Ranganath and A. Ng},
  booktitle={International Conference on Machine Learning},
There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max… 

Learning deep generative models

The aim of the thesis is to demonstrate that deep generative models that contain many layers of latent variables and millions of parameters can be learned efficiently, and that the learned high-level feature representations can be successfully applied in a wide spectrum of application domains, including visual object recognition, information retrieval, and classification and regression tasks.

Learning with Hierarchical-Deep Models

Efficient learning and inference algorithms for the HDP-DBM model are presented and it is shown that it is able to learn new concepts from very few examples on CIFAR-100 object recognition, handwritten character recognition, and human motion capture datasets.

Learning Deep Representations, Embeddings and Codes from the Pixel Level of Natural and Medical Images

This thesis considers the task of learning two specific types of image representations from standard size RGB images: a semi-supervised dense low-dimensional embedding and an unsupervised sparse binary code and introduces a new algorithm called the deep matching pursuit network (DMP) that efficiently learns features layer-by-layer from the pixel level without the need for backpropagation fine tuning.

Hierarchical Convolutional Deep Learning in Computer Vision

This work introduces a new regularization method for convolutional networks called stochastic pooling which relies on sampling noise to prevent these powerful models from overfitting, and introduces a novel optimization method called ADADELTA which shows promising convergence speeds in practice whilst being robust to hyper-parameter selection.

A probabilistic model for recursive factorized image features

A probabilistic model that learns and infers all layers of the hierarchy jointly and presents a novel generative model based on Latent Dirichlet Allocation to this end, outperforming existing hierarchical approaches and demonstrating performance on par with current single-feature state-of-the-art models.

A Deep Generative Deconvolutional Image Model

A deep generative model is developed for representation and analysis of images, based on a hierarchical convolutional dictionary-learning framework, demonstrating that the proposed model achieves results that are highly competitive with similarly sized Convolutional neural networks.

Detection guided deconvolutional network for hierarchical feature learning

Temporal Continuity Learning for Convolutional Deep Belief Networks

The goal of this work is to develop a computer algorithm which can replicate this sort of learning called temporal continuity learning, which uses Deep Belief Networks and entirely different heuristics to measure how ’good’ a representation is.

Deep Predictive Coding Networks

Deep predictive coding networks is proposed, a hierarchical generative model that empirically alters priors on the latent representations in a dynamic and context-sensitive manner that captures the temporal dependencies in time-varying signals and uses top-down information to modulate the representation in lower layers.

Robust Visual Recognition Using Multilayer Generative Neural Networks

This thesis develops a probabilistic denoising algorithm to determine a subset of the hidden layer nodes to unclamp and shows significantly better performance over the standard DBN implementations for various sources of noise on the standard and Variations MNIST databases.



A Fast Learning Algorithm for Deep Belief Nets

A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.

Sparse deep belief net model for visual area V2

An unsupervised learning model is presented that faithfully mimics certain properties of visual area V2 and the encoding of these more complex "corner" features matches well with the results from the Ito & Komatsu's study of biological V2 responses, suggesting that this sparse variant of deep belief networks holds promise for modeling more higher-order features.

Large-scale deep unsupervised learning using graphics processors

It is argued that modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods.

Greedy Layer-Wise Training of Deep Networks

These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.

Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition

An unsupervised method for learning a hierarchy of sparse feature detectors that are invariant to small shifts and distortions that alleviates the over-parameterization problems that plague purely supervised learning procedures, and yields good performance with very few labeled training samples.

Efficient Learning of Sparse Representations with an Energy-Based Model

A novel unsupervised method for learning sparse, overcomplete features using a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector.

Emergence of simple-cell receptive field properties by learning a sparse code for natural images

It is shown that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex.

Deep Learning with Kernel Regularization for Visual Recognition

A novel regularization method is proposed that takes advantage of kernel methods, where an oracle kernel function represents prior knowledge about the recognition task of interest.

Hierarchical Bayesian inference in the visual cortex.

  • T. LeeD. Mumford
  • Biology, Computer Science
    Journal of the Optical Society of America. A, Optics, image science, and vision
  • 2003
This work proposes a new theoretical setting based on the mathematical framework of hierarchical Bayesian inference for reasoning about the visual system, and suggests that the algorithms of particle filtering and Bayesian-belief propagation might model these interactive cortical computations.