Corpus ID: 1808153

Measuring Invariances in Deep Networks

@inproceedings{Goodfellow2009MeasuringII,
  title={Measuring Invariances in Deep Networks},
  author={Ian J. Goodfellow and Quoc V. Le and Andrew M. Saxe and Honglak Lee and A. Ng},
  booktitle={NIPS},
  year={2009}
}
For many pattern recognition tasks, the ideal input feature would be invariant to multiple confounding properties (such as illumination and viewing angle, in computer vision applications). Recently, deep architectures trained in an unsupervised manner have been proposed as an automatic method for extracting useful features. However, it is difficult to evaluate the learned features by any means other than using them in a classifier. In this paper, we propose a number of empirical tests that… 
Locally Scale-Invariant Convolutional Neural Networks
TLDR
A simple model is presented that allows ConvNets to learn features in a locally scale-invariant manner without increasing the number of model parameters, and is shown on a modified MNIST dataset that when faced with scale variation, building in scale-Invariance allows Conv net to learn more discriminative features with reduced chances of over-fitting.
Learning hierarchical representations for face verification
TLDR
It is shown that a recognition system using only representations obtained from deep learning can achieve comparable accuracy with a system using a combination of hand-crafted image descriptors, and empirically show that learning weights not only is necessary for obtaining good multilayer representations, but also provides robustness to the choice of the network architecture parameters.
Learning hierarchical representations for face verification with convolutional deep belief networks
TLDR
It is shown that a recognition system using only representations obtained from deep learning can achieve comparable accuracy with a system using a combination of hand-crafted image descriptors, and empirically show that learning weights not only is necessary for obtaining good multilayer representations, but also provides robustness to the choice of the network architecture parameters.
Is Robustness To Transformations Driven by Invariant Neural Representations?
TLDR
The results with state-of-the-art DCNNs indicate that invariant representations strengthen as the number of transformed categories in the training set is increased, and are much more prominent with local transformations such as blurring and high-pass filtering, compared to geometric transformations that entail changes in the spatial arrangement of the object.
Towards Distortion-Predictable Embedding of Neural Networks
TLDR
This work proposes a new loss function, derived from the contrastive loss, that creates models with more predicable mappings and also quantifies distortions, and makes a step towards embeddings where features of distorted inputs are related and can be derived from each others by the intensity of the distortion.
Understanding Representations Learned in Deep Architectures
Deep architectures have demonstrated state-of-the-art pe rformance in a variety of settings, especially with vision datasets. Deep learning a lgorithms are based on learning several levels of
Unsupervised Rotation Factorization in Restricted Boltzmann Machines
TLDR
This paper presents an extended novel RBM that learns rotation invariant features by explicitly factorizing for rotation nuisance in 2D image inputs within an unsupervised framework and shows that this method outperforms the current state-of-the-art RBM approaches on three different benchmark datasets.
Major in Information Technologies Measuring Robustness of Classifiers to Geometric Transformations Student
For many classification tasks, the ideal classifier should be invariant to geometric transformations such as changing the view angle. However, this cannot be said decisively for the state-of-the-art
Manitest: Are classifiers really invariant?
TLDR
The Manitest method is proposed, built on the efficient Fast Marching algorithm to compute the invariance of classifiers, which quantifies in particular the importance of data augmentation for learning invariance from data, and the increased invariances of convolutional neural networks with depth.
Transform-Invariant Convolutional Neural Networks for Image Classification and Search
TLDR
This paper proposes randomly transforming feature maps of CNNs during the training stage to prevent complex dependencies of specific rotation, scale, and translation levels of training images in CNN models and shows that random transformation provides significant improvements ofCNNs on many benchmark tasks, including small-scaleimage recognition, large-scale image recognition, and image retrieval.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 33 REFERENCES
Sparse Feature Learning for Deep Belief Networks
TLDR
This work proposes a simple criterion to compare and select different unsupervised machines based on the trade-off between the reconstruction error and the information content of the representation, and describes a novel and efficient algorithm to learn sparse representations.
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
TLDR
An unsupervised method for learning a hierarchy of sparse feature detectors that are invariant to small shifts and distortions that alleviates the over-parameterization problems that plague purely supervised learning procedures, and yields good performance with very few labeled training samples.
Scaling learning algorithms towards AI
TLDR
It is argued that deep architectures have the potential to generalize in non-local ways, i.e., beyond immediate neighbors, and that this is crucial in order to make progress on the kind of complex tasks required for artificial intelligence.
Sparse deep belief net model for visual area V2
TLDR
An unsupervised learning model is presented that faithfully mimics certain properties of visual area V2 and the encoding of these more complex "corner" features matches well with the results from the Ito & Komatsu's study of biological V2 responses, suggesting that this sparse variant of deep belief networks holds promise for modeling more higher-order features.
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
TLDR
The convolutional deep belief network is presented, a hierarchical generative model which scales to realistic image sizes and is translation-invariant and supports efficient bottom-up and top-down probabilistic inference.
Learning methods for generic object recognition with invariance to pose and lighting
  • Yann LeCun, F. Huang, L. Bottou
  • Computer Science
    Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004.
  • 2004
TLDR
A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second and proved impractical, while convolutional nets yielded 16/7% error.
A Fast Learning Algorithm for Deep Belief Nets
TLDR
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
Slow Feature Analysis: Unsupervised Learning of Invariances
TLDR
Slow feature analysis (SFA) is a new method for learning invariant or slowly varying features from a vectorial input signal that is guaranteed to find the optimal solution within a family of functions directly and can learn to extract a large number of decor-related features, which are ordered by their degree of invariance.
An empirical evaluation of deep architectures on problems with many factors of variation
TLDR
A series of experiments indicate that these models with deep architectures show promise in solving harder learning problems that exhibit many factors of variation.
Greedy Layer-Wise Training of Deep Networks
TLDR
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
...
1
2
3
4
...