Corpus ID: 211010953

On Implicit Regularization in $\beta$-VAEs

  title={On Implicit Regularization in \$\beta\$-VAEs},
  author={Abhishek Kumar and Ben Poole},
  journal={arXiv: Learning},
While the impact of variational inference (VI) on posterior inference in a fixed generative model is well-characterized, its role in regularizing a learned generative model when used in variational autoencoders (VAEs) is poorly understood. We study the regularizing effects of variational distributions on learning in generative models from two perspectives. First, we analyze the role that the choice of variational family plays in imparting uniqueness to the learned model by restricting the set… Expand
Regularized linear autoencoders recover the principal components, eventually
It is shown that when trained with proper regularization, LAEs can directly learn the optimal representation -- ordered, axis-aligned principal components. Expand
Isometric Autoencoders
This paper advocates an isometry (i.e., local distance preserving) regularizer that fixes both intrinsic and extrinsic degrees of freedom and provide a non-linear generalization to principal component analysis (PCA). Expand
Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
Sebastian Houben1, Stephanie Abrecht2, Maram Akila1, Andreas Bär15, Felix Brockherde10, Patrick Feifel8, Tim Fingscheidt15, Sujan Sai Gannamaneni1, Seyed Eghbal Ghobadi8, Ahmed Hammam8, AnselmExpand
Rate-Regularization and Generalization in VAEs
It is shown that generalization performance continues to improve even after the mutual information saturates, indicating that the gap on the bound affects generalization, suggesting that the standard spherical Gaussian prior is not an inductive bias that typically improves generalization. Expand


Variational Laplace Autoencoders
A general framework named Variational Laplace Autoencoders (VLAEs) for training deep generative models based on the Laplace approximation of the latent variable posterior, VLAEs enhance the expressiveness of the posterior while reducing the amortization error. Expand
From Variational to Deterministic Autoencoders
It is shown, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules. Expand
An Information-Theoretic Analysis of Deep Latent-Variable Models
An information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference and how this framework sheds light on many recent proposed extensions to the variational autoencoder family is presented. Expand
Regularized Autoencoders via Relaxed Injective Probability Flow
A generative model based on probability flows that does away with the bijectivity requirement on the model and only assumes injectivity is proposed, which provides another perspective on regularized autoencoders (RAEs), with the final objectives resembling RAEs with specific regularizers that are derived by lower bounding the probability flow objective. Expand
Amortized Inference Regularization
This paper challenges the belief that amortized inference is simply a mechanism for approximating maximum likelihood training and illustrates that regularization of the amortization family provides a new direction for understanding and improving generalization in VAEs. Expand
beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework
Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificialExpand
Variational Autoencoders and Nonlinear ICA: A Unifying Framework
This work shows that for a broad family of deep latent-variable models, identification of the true joint distribution over observed and latent variables is actually possible up to very simple transformations, thus achieving a principled and powerful form of disentanglement. Expand
Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations
It is demonstrated that the proposed prior significantly mitigates the trade-off between reconstruction loss and disentanglement over the state of the art and resolves the problem of unidentifiability of the standard VAE normal prior. Expand
Fixing a Broken ELBO
This framework derives variational lower and upper bounds on the mutual information between the input and the latent variable, and uses these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Expand
Understanding disentangling in $\beta$-VAE
A modification to the training regime of $\ beta$-VAE is proposed, that progressively increases the information capacity of the latent code during training, to facilitate the robust learning of disentangled representations in $\beta$- VAE, without the previous trade-off in reconstruction accuracy. Expand