Reducing the Dimensionality of Data with Neural Networks

  title={Reducing the Dimensionality of Data with Neural Networks},
  author={Geoffrey E. Hinton and Ruslan Salakhutdinov},
  pages={504 - 507}
High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than… 

Dimensionality Reduction Using Neural Networks

Deep autoencoders showed signs of improvement when pretrained over the ones without pretraining, and the tuning which followed the pretraining approach was able to reduce the data dimensionality very efficiently.

A Deep Neural Network Architecture Using Dimensionality Reduction with Sparse Matrices

A new deep neural network architecture is presented, motivated by sparse random matrix theory that uses a low-complexity embedding through a sparse matrix instead of a conventional stacked autoencoder, demonstrating experimentally that classification performance does not deteriorate if the autoenCoder is replaced with a computationally-efficient sparse dimensionality reduction matrix.

Index-learning of unsupervised low dimensional embeddings

A simple unsupervised learning method for creating low-dimensional embeddings on datasets where creating a low dimensional representation requires throwing away so much information that it is unreasonable to attempt to reconstruct the input from any kind of lowdimensional representation.

Dimensionality Reduction Applied to Time Response of Linear Systems Using Autoencoders

Results show that is possible to use a deep autoencoder to capture the dynamical behavior of a dynamical system in its latent layer and to perform a compact representation of the time response of linear systems.

Self-supervised Dimensionality Reduction with Neural Networks and Pseudo-labeling

This work proposes a deep learning DR method called Self-Supervised Network Projection (SSNP) which does DR based on pseudo-labels obtained from clustering, and shows that SSNP produces better cluster separation than autoencoders, has out-of-sample, inverse mapping, and clustering capabilities, and is very fast and easy to use.

Dimensionality compression and expansion in Deep Neural Networks

This work contributes by shedding light on the success of deep neural networks in disentangling data in high-dimensional space while achieving good generalization, and invites new learning strategies focused on optimizing measurable geometric properties of learned representations, beginning with their intrinsic dimensionality.

From Principal Subspaces to Principal Components with Linear Autoencoders

This paper shows how to recover the loading vectors from the autoencoder weights, which are not identical to the principal component loading vectors.

Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network

A new type of stochastic neural network is proposed under a rigorous probabilistic framework and it is shown that it can be used for sufficient dimension reduction for large-scale data.

Random Projection in Deep Neural Networks

This work investigates the ways in which deep learning methods can benefit from random projection (RP), a classic linear dimensionality reduction method. We focus on two areas where, as we have


A multi-layer neural network with multiple hidden layers was trained as an autoencoder using steepest descent, scaled conjugate gradient and alopex algorithms and results indicate that while pretraining is important for obtaining good results, the pretraining approach used by Hinton et al. obtains lower RMSE than other methods.



Dimension Reduction by Local Principal Component Analysis

A local linear approach to dimension reduction that provides accurate representations and is fast to compute is developed and it is shown that the local linear techniques outperform neural network implementations.

Nonlinear dimensionality reduction by locally linear embedding.

Locally linear embedding (LLE) is introduced, an unsupervised learning algorithm that computes low-dimensional, neighborhood-preserving embeddings of high-dimensional inputs that learns the global structure of nonlinear manifolds.

A Fast Learning Algorithm for Deep Belief Nets

A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.

Replicator neural networks for universal optimal source coding.

A theorem shows that a class of replicator networks can, through the minimization of mean squared reconstruction error, carry out optimal data compression for arbitrary data vector sources.

Neural networks and physical systems with emergent collective computational abilities.

  • J. Hopfield
  • Computer Science
    Proceedings of the National Academy of Sciences of the United States of America
  • 1982
A model of a system having a large number of simple equivalent components, based on aspects of neurobiology but readily adapted to integrated circuits, produces a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size.

Neural Computation

The nervous system is able to develop by combining on one hand a only limited amount of genetic information and, on the other hand, the input it receives, and it might be possible to develop a brain from there.

Machine learning

Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis.

Parallel Distributed Processing Volume 1: Foundations

1,227,914. A die for laminated films. NORTHERN PETROCHEMICAL CO. 25 April, 1969 [26 April, 1968], No. 21298/69. Heading B5B. [Also in Division F2] A die for the extrusion of laminated plastic tubing

References and Notes

our experimentation could eventually be used to discredit our findings, should they happen not to agree with the original observations. It seems important that all experiments in the rapidly

IL-13受体α2降低血吸虫病肉芽肿的炎症反应并延长宿主存活时间[英]/Mentink-Kane MM,Cheever AW,Thompson RW,et al//Proc Natl Acad Sci U S A