Corpus ID: 20282961

Neural Discrete Representation Learning

@inproceedings{Oord2017NeuralDR,
  title={Neural Discrete Representation Learning},
  author={A{\"a}ron van den Oord and Oriol Vinyals and Koray Kavukcuoglu},
  booktitle={NIPS},
  year={2017}
}
Learning useful representations without supervision remains a key challenge in machine learning. [...] Key Method In order to learn a discrete latent representation, we incorporate ideas from vector quantisation (VQ). Using the VQ method allows the model to circumvent issues of "posterior collapse" -- where the latents are ignored when they are paired with a powerful autoregressive decoder -- typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can…Expand
Robust Training of Vector Quantized Bottleneck Models
TLDR
This paper focuses on VQ-VAE, a state-of-the-art discrete bottleneck model shown to perform on par with its continuous counterparts, and shows that the codebook learning can suffer from poor initialization and non-stationarity of clustered encoder outputs, but can be successfully overcome by increasing the learning rate. Expand
Depthwise Discrete Representation Learning
TLDR
This work proposes applying VQ along the feature axis and hypothesizes that by doing so, it is learning a mapping between the codebook vectors and the marginal distribution of the prior feature space. Expand
Towards a better understanding of Vector Quantized Autoencoders
TLDR
This work investigates an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm, and develops a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference. Expand
Theory and Experiments on Vector Quantized Autoencoders
TLDR
This work investigates an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm, and develops a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference. Expand
Supervised Vector Quantized Variational Autoencoder for Learning Interpretable Global Representations
TLDR
It is proved that S-VQ-VAE was able to learn the global genetic characteristics of samples perturbed by the same class of perturbagen (PCL), and further revealed the mechanism correlations between PCLs. Expand
Vector-Quantized Autoregressive Predictive Coding
TLDR
This work proposes Vector-Quantized Autoregressive Predictive Coding (VQ-APC), a novel model that produces quantized representations, allowing us to explicitly control the amount of information encoded in the representations, and finds that there exists a point where phonetic and speaker information are amplified to maximize a self-supervised objective. Expand
Unsupervised Discrete Representation Learning
TLDR
This work proposes a method called Information Maximizing Self-Augmented Training (IMSAT), which uses data augmentation to impose the invariance on discrete representations and encourages the predicted representations of augmented data points to be close to those of the original data points in an end-to-end fashion. Expand
Discrete Infomax Codes for Meta-Learning
TLDR
This paper presents a method for learning a stochastic encoder that yields discrete p-way codes of length d by maximizing the mutual information between representations and labels, and shows that using shorter codes reduces overfitting in the context of few-shot classification. Expand
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
TLDR
This work proposes DeCoAR 2.0, a Deep Contextualized Acoustic Representation with vector quantization, which uses Transformers in encoding module instead of LSTMs and proposes an objective that combines the reconstructive loss withvector quantization diversity loss to train speech representations. Expand
Associative Compression Networks for Representation Learning
TLDR
It is concluded that ACNs are a promising new direction for representation learning: one that steps away from IID modelling, and towards learning a structured description of the dataset as a whole. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 43 REFERENCES
Variational Lossy Autoencoder
TLDR
This paper presents a simple but principled method to learn global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN with greatly improve generative modeling performance of VAEs. Expand
PixelVAE: A Latent Variable Model for Natural Images
Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficultyExpand
Deep AutoRegressive Networks
TLDR
An efficient approximate parameter estimation method based on the minimum description length (MDL) principle is derived, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. Expand
Importance Weighted Autoencoders
TLDR
The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks. Expand
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
TLDR
This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations. Expand
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
TLDR
Experiments show that InfoGAN learns interpretable representations that are competitive with representations learned by existing fully supervised methods. Expand
Generative Adversarial Nets
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and aExpand
Auto-Encoding Variational Bayes
TLDR
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced. Expand
WaveNet: A Generative Model for Raw Audio
TLDR
WaveNet, a deep neural network for generating raw audio waveforms, is introduced; it is shown that it can be efficiently trained on data with tens of thousands of samples per second of audio, and can be employed as a discriminative model, returning promising results for phoneme recognition. Expand
Generating Sentences from a Continuous Space
TLDR
This work introduces and study an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences that allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features. Expand
...
1
2
3
4
5
...