• Corpus ID: 1327363

Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

@article{Dilokthanakul2016DeepUC,
  title={Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders},
  author={Nat Dilokthanakul and Pedro A. M. Mediano and Marta Garnelo and M. J. Lee and Hugh Salimbeni and Kai Arulkumaran and Murray Shanahan},
  journal={ArXiv},
  year={2016},
  volume={abs/1611.02648}
}
We study a variant of the variational autoencoder model (VAE) with a Gaussian mixture as a prior distribution, with the goal of performing unsupervised clustering through deep generative models. We observe that the known problem of over-regularisation that has been shown to arise in regular VAEs also manifests itself in our model and leads to cluster degeneracy. We show that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied… 

Figures and Tables from this paper

Unsupervised Clustering through Gaussian Mixture Variational AutoEncoder with Non-Reparameterized Variational Inference and Std Annealing
TLDR
This paper improves the performance of Gaussian Mixture VAE, by optimizing it with a Monte Carlo objective (including the q(z|y, x) term), with non-reparameterized Variational Inference for Monte Carlo Objectives (VIMCO) method.
Variational Clustering: Leveraging Variational Autoencoders for Image Clustering
TLDR
This work is the first to pursue image clustering using VAEs in a purely unsupervised manner on real image datasets, and proposes a novel reparametrization of the latent space consisting of a mixture of discrete and continuous variables.
Truncated Gaussian-Mixture Variational AutoEncoder
TLDR
A novel generative process is proposed, in which a Gaussian-mixture is used to model a few major clusters in the data, and use a non-informative uniform distribution to capture the remaining data, called tGM-VAE.
Deep Clustering Based On A Mixture Of Autoencoders
TLDR
A deep clustering algorithm based on a mixture of deep autoencoders where each cluster is represented by an autoencoder, which shows significant improvement over state-of-the-art methods on image and text corpora.
Disentangling to Cluster: Gaussian Mixture Variational Ladder Autoencoders
TLDR
This work proposes a clustering algorithm, VLAC, that outperforms a Gaussian Mixture DGM in cluster accuracy over digit identity on the test set of SVHN and demonstrates learning clusters jointly over numerous layers of the hierarchy of latent variables for the data.
Hierarchical Decompositional Mixtures of Variational Autoencoders
TLDR
This work proposes a novel hierarchical mixture model over low-dimensional VAE experts, which decomposes the overall learning problem into many smaller problems, which are coordinated by the hierarchical mixture, represented by a sum-product network.
Latent Tree Variational Autoencoder for Joint Representation Learning and Multidimensional Clustering
TLDR
This work proposes latent tree variational autoencoder (LTVAE), which simultaneously performs representation learning and multidimensional clustering, and achieves state-of-the-art clustering performance and reals reasonable multifacet structures of the data.
Deep Spectral Clustering Using Dual Autoencoder Network
TLDR
A joint learning framework for discriminative embedding and spectral clustering is proposed, which can significantly outperform state-of-the-art clustering approaches and be more robust to noise.
Deep Mixture Generative Autoencoders.
  • Fei Ye, A. Bors
  • Computer Science
    IEEE transactions on neural networks and learning systems
  • 2021
TLDR
This article extends the classical concept of Gaussian mixtures into the deep variational framework by proposing a mixture of VAEs (MVAE), where each component in the MVAE model is implemented by a variational encoder and has an associated subdecoder.
Deep Conditional Gaussian Mixture Model for Constrained Clustering
TLDR
This work proposes a novel framework for constrained clustering that is intuitive, interpretable, and can be trained efficiently in the framework of stochastic gradient variational inference, by explicitly integrating domain knowledge in the form of probabilistic relations.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 38 REFERENCES
Ladder Variational Autoencoders
TLDR
A new inference model is proposed, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network.
How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks
TLDR
This work proposes three advances in training algorithms of variational autoencoders, for the first time allowing to train deep models of up to five stochastic layers, using a structure similar to the Ladder network as the inference model and shows state-of-the-art log-likelihood results for generative modeling on several benchmark datasets.
Auto-Encoding Variational Bayes
TLDR
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
Variational Lossy Autoencoder
TLDR
This paper presents a simple but principled method to learn global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN with greatly improve generative modeling performance of VAEs.
Early Visual Concept Learning with Unsupervised Deep Learning
TLDR
An unsupervised approach for learning disentangled representations of the underlying factors of variation by applying the same learning pressures as have been suggested to act in the ventral visual stream in the brain is proposed.
Improved Variational Inference with Inverse Autoregressive Flow
TLDR
A new type of normalizing flow, inverse autoregressive flow (IAF), is proposed that, in contrast to earlier published flows, scales well to high-dimensional latent spaces and significantly improves upon diagonal Gaussian approximate posteriors.
Unsupervised Deep Embedding for Clustering Analysis
TLDR
Deep Embedded Clustering is proposed, a method that simultaneously learns feature representations and cluster assignments using deep neural networks and learns a mapping from the data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective.
Tagger: Deep Unsupervised Perceptual Grouping
TLDR
This work presents a framework for efficient perceptual inference that explicitly reasons about the segmentation of its inputs and features and greatly improves on the semi-supervised result of a baseline Ladder network on the authors' dataset, indicating that segmentation can also improve sample efficiency.
Adam: A Method for Stochastic Optimization
TLDR
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Stochastic Backpropagation through Mixture Density Distributions
TLDR
Together with the reparameterization trick applied to the individual mixture components, this estimator makes it straightforward to train variational autoencoders with mixture-distributed latent variables, or to perform stochastic variational inference with a mixture density variational posterior.
...
1
2
3
4
...