Learning Representations by Maximizing Mutual Information in Variational Autoencoders

@article{LotfiRezaabad2020LearningRB,
  title={Learning Representations by Maximizing Mutual Information in Variational Autoencoders},
  author={Ali Lotfi-Rezaabad and Sriram Vishwanath},
  journal={2020 IEEE International Symposium on Information Theory (ISIT)},
  year={2020},
  pages={2729-2734}
}
Variational autoencoders (VAE) have ushered in an new era of unsupervised learning methods for complex distributions. Although these techniques are elegant in their approach, they are typically not useful for representation learning. In this work, we propose a simple yet powerful class of VAEs that simultaneously result in meaningful learned representations. Our solution is to combine traditional VAEs with mutual information maximization, with the goal to enhance amortized inference in VAEs… 
Deep clustering by maximizing mutual information in variational auto-encoder
TLDR
It is shown that the proposed model can significantly boost the performance of clustering by learning effective and reliable representations for downstream machine learning tasks and is competitive with existing state-of-the-arts on multiple performance metrics.
On Disentanglement and Mutual Information in Semi-Supervised Variational Auto-Encoders
TLDR
This abstract considers the semisupervised setting, in which the factors of variation are labelled for a small fraction of the authors' samples, and examines how the quality of learned representations is affected by the dimension of the unsupervised component of the latent space.
Discriminative Multimodal Learning via Conditional Priors in Generative Models
TLDR
This research introduces a novel conditional multi-modal discriminative model that uses an informative prior distribution and optimizes a likelihood-free objective function that maximizes mutual information between joint representations and missing modalities.
Category-Learning with Context-Augmented Autoencoder
TLDR
This work trains a Variational Autoencoder in such a way, that it makes transformation outcome predictable by auxiliary network in terms of the hidden representation, and believes that the classification accuracy of a linear classifier on the learned representation is a good metric to measure its interpretability.
Deep Dive into Semi-Supervised ELBO for Improving Classification Performance
TLDR
It is shown that mutual information between input and class labels decreases during maximization of ELBO objective, and a method is proposed to address this issue and enforce cluster assumption to aid in classification.
Learning Robust Representations via Multi-View Information Bottleneck
TLDR
A theoretical analysis leads to the definition of a new multi-View model which produces state-of-the-art results on two standard multi-view datasets, Sketchy and MIR-Flickr, empirically showing better generalization capabilities when compared to traditional unsupervised approaches.
MULTI-VIEW INFORMATION BOTTLENECK
The information bottleneck method Tishby et al. (2000) provides an informationtheoretic method for representation learning, by training an encoder to retain all information which is relevant for
Mutual Information Constraints for Monte-Carlo Objectives
TLDR
This work constructs estimators of the Kullback-Leibler divergence of the true posterior from the prior by recycling samples used in the objective, with which to train models of continuous and discrete latents at much improved rate-distortion and no posterior collapse.
Unsupervised Outlier Detection via Transformation Invariant Autoencoder
TLDR
To mitigate the negative effect of noise introduced by outliers and stabilize the network training, the most confident inliers likely examples in each epoch are selected as the training set by incorporating adaptive self-paced learning in the TIAE framework.
Denoising Normalizing Flow
Normalizing flows (NF) are expressive as well as tractable density estimation methods whenever the support of the density is diffeomorphic to the entire dataspace. However, real-world data sets
...
1
2
...

References

SHOWING 1-10 OF 29 REFERENCES
InfoVAE: Balancing Learning and Inference in Variational Autoencoders
TLDR
It is shown that the proposed Info-VAE model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution.
Learning deep representations by mutual information estimation and maximization
TLDR
It is shown that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks and is an important step towards flexible formulations of representation learning objectives for specific end-goals.
Recent Advances in Autoencoder-Based Representation Learning
TLDR
An in-depth review of recent advances in representation learning with a focus on autoencoder-based models and makes use of meta-priors believed useful for downstream tasks, such as disentanglement and hierarchical organization of features.
Variational Lossy Autoencoder
TLDR
This paper presents a simple but principled method to learn global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN with greatly improve generative modeling performance of VAEs.
Spherical Latent Spaces for Stable Variational Autoencoders
TLDR
This work experiments with another choice of latent distribution, namely the von Mises-Fisher (vMF) distribution, which places mass on the surface of the unit hypersphere and shows that they learn richer and more nuanced structures in their latent representations than their Gaussian counterparts.
beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework
Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial
Importance Weighted Autoencoders
TLDR
The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks.
Fixing a Broken ELBO
TLDR
This framework derives variational lower and upper bounds on the mutual information between the input and the latent variable, and uses these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy.
Adversarially Regularized Autoencoders
TLDR
This work proposes a flexible method for training deep latent variable models of discrete structures based on the recently-proposed Wasserstein autoencoder (WAE), and shows that the latent representation can be trained to perform unaligned textual style transfer, giving improvements both in automatic/human evaluation compared to existing methods.
Neural Discrete Representation Learning
TLDR
Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations.
...
1
2
3
...