Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

@inproceedings{Lee2009ConvolutionalDB,
  title={Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations},
  author={Honglak Lee and Roger B. Grosse and Rajesh Ranganath and A. Ng},
  booktitle={ICML '09},
  year={2009}
}
There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max… 
Learning deep generative models
TLDR
The aim of the thesis is to demonstrate that deep generative models that contain many layers of latent variables and millions of parameters can be learned efficiently, and that the learned high-level feature representations can be successfully applied in a wide spectrum of application domains, including visual object recognition, information retrieval, and classification and regression tasks.
Learning with Hierarchical-Deep Models
TLDR
Efficient learning and inference algorithms for the HDP-DBM model are presented and it is shown that it is able to learn new concepts from very few examples on CIFAR-100 object recognition, handwritten character recognition, and human motion capture datasets.
Learning Deep Representations, Embeddings and Codes from the Pixel Level of Natural and Medical Images
TLDR
This thesis considers the task of learning two specific types of image representations from standard size RGB images: a semi-supervised dense low-dimensional embedding and an unsupervised sparse binary code and introduces a new algorithm called the deep matching pursuit network (DMP) that efficiently learns features layer-by-layer from the pixel level without the need for backpropagation fine tuning.
Hierarchical Convolutional Deep Learning in Computer Vision
TLDR
This work introduces a new regularization method for convolutional networks called stochastic pooling which relies on sampling noise to prevent these powerful models from overfitting, and introduces a novel optimization method called ADADELTA which shows promising convergence speeds in practice whilst being robust to hyper-parameter selection.
A probabilistic model for recursive factorized image features
TLDR
A probabilistic model that learns and infers all layers of the hierarchy jointly and presents a novel generative model based on Latent Dirichlet Allocation to this end, outperforming existing hierarchical approaches and demonstrating performance on par with current single-feature state-of-the-art models.
A Novel Sparse Deep Belief Network for Unsupervised Feature Learning
TLDR
A novel version of sparse deep belief network for unsupervised feature extraction that learns hierarchical representations which mimicks computations in the cortical hierarchy, and obtains more discriminative representation than PCA and several basic algorithms of deep belief networks.
A Deep Generative Deconvolutional Image Model
TLDR
A deep generative model is developed for representation and analysis of images, based on a hierarchical convolutional dictionary-learning framework, demonstrating that the proposed model achieves results that are highly competitive with similarly sized Convolutional neural networks.
Detection guided deconvolutional network for hierarchical feature learning
TLDR
Experiments demonstrate that the proposed detection-guided hierarchical learning algorithm to train a deep deconvolutional network outperforms some recent hierarchical feature descriptors as well as classical hand-crafted features on the task of object recognition.
Learning to Learn with Compound HD Models
TLDR
Efficient learning and inference algorithms for the HDP-DBM model are presented and it is shown that it is able to learn new concepts from very few examples on CIFAR-100 object recognition, handwritten character recognition, and human motion capture datasets.
Temporal Continuity Learning for Convolutional Deep Belief Networks
The human visual system can robustly recognize objects, even though a single object can project many different images onto the retina. Furthermore, humans learn to perform this task from mostly
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 32 REFERENCES
A Fast Learning Algorithm for Deep Belief Nets
TLDR
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
Sparse deep belief net model for visual area V2
TLDR
An unsupervised learning model is presented that faithfully mimics certain properties of visual area V2 and the encoding of these more complex "corner" features matches well with the results from the Ito & Komatsu's study of biological V2 responses, suggesting that this sparse variant of deep belief networks holds promise for modeling more higher-order features.
Large-scale deep unsupervised learning using graphics processors
TLDR
It is argued that modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods.
Greedy Layer-Wise Training of Deep Networks
TLDR
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories
TLDR
The incremental algorithm is compared experimentally to an earlier batch Bayesian algorithm, as well as to one based on maximum-likelihood, which have comparable classification performance on small training sets, but incremental learning is significantly faster, making real-time learning feasible.
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
TLDR
An unsupervised method for learning a hierarchy of sparse feature detectors that are invariant to small shifts and distortions that alleviates the over-parameterization problems that plague purely supervised learning procedures, and yields good performance with very few labeled training samples.
Efficient Learning of Sparse Representations with an Energy-Based Model
TLDR
A novel unsupervised method for learning sparse, overcomplete features using a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector.
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
TLDR
It is shown that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex.
Deep Learning with Kernel Regularization for Visual Recognition
TLDR
A novel regularization method is proposed that takes advantage of kernel methods, where an oracle kernel function represents prior knowledge about the recognition task of interest.
Hierarchical Bayesian inference in the visual cortex.
  • T. Lee, D. Mumford
  • Computer Science, Medicine
    Journal of the Optical Society of America. A, Optics, image science, and vision
  • 2003
TLDR
This work proposes a new theoretical setting based on the mathematical framework of hierarchical Bayesian inference for reasoning about the visual system, and suggests that the algorithms of particle filtering and Bayesian-belief propagation might model these interactive cortical computations.
...
1
2
3
4
...