# Sinkhorn AutoEncoders

@article{Patrini2019SinkhornA, title={Sinkhorn AutoEncoders}, author={Giorgio Patrini and Marcello Carioni and Patrick Forr'e and Samarth Bhargav and Max Welling and Rianne van den Berg and Tim Genewein and Frank Nielsen}, journal={ArXiv}, year={2019}, volume={abs/1810.01118} }

Optimal transport offers an alternative to maximum likelihood for learning generative autoencoding models. We show that minimizing the p-Wasserstein distance between the generator and the true data distribution is equivalent to the unconstrained min-min optimization of the p-Wasserstein distance between the encoder aggregated posterior and the prior in latent space, plus a reconstruction error. We also identify the role of its trade-off hyperparameter as the capacity of the generator: its… Expand

#### 35 Citations

Generative Modeling with Optimal Transport Maps

- Computer Science
- ArXiv
- 2021

A minmax optimization algorithm is derived to efficiently compute OT maps for the quadratic cost (Wasserstein-2 distance) and this approach is extended to the case when the input and output distributions are located in the spaces of different dimensions and derive error bounds for the computed OT map. Expand

WiSE-ALE: Wide Sample Estimator for Approximate Latent Embedding

- Computer Science, Mathematics
- 2019

This paper derives an alternative variational lower bound from the one common in VAEs, and demonstrates that WiSE-ALE can reach excellent reconstruction quality in comparison to other state-of-the-art VAE models, while still retaining the ability to learn a smooth, compact representation. Expand

Dual Rejection Sampling for Wasserstein Auto-Encoders

- Computer Science
- ECAI
- 2020

A novel dual rejection sampling method is proposed to improve the performance of WAE on the generated samples in the sampling phase and corrects the generative prior by a discriminator based rejection sampling scheme in latent space and then rectifies the generated distribution by another discriminatorbased rejection sampling technique in data space. Expand

Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE

- Computer Science
- ArXiv
- 2021

This paper shows that using the contrastive learning framework to optimize the WAE loss achieves faster convergence and more stable optimization compared with existing popular algorithms for WAE. Expand

Learning from Nested Data with Ornstein Auto-Encoders

- Computer Science
- ICML
- 2021

The product-space OAE (PSOAE) is presented that minimizes a tighter upper bound of the distance and achieves orthogonality in the representation space and alleviates the instability of RIOAE and provides more flexible representation of nested data. Expand

AUTOENCODERS WITH SPHERICAL SLICED FUSED

- 2021

Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing a reconstruction loss together with a relational regularization on the latent space. A recent… Expand

WiSE-VAE: Wide Sample Estimator VAE

- Mathematics, Computer Science
- ArXiv
- 2019

This paper derives an alternative variational lower bound from the one common in VAEs, and demonstrates that WiSE-VAE can reach excellent reconstruction quality in comparison to other state-of-the-art VAE models, while still retaining the ability to learn a smooth, compact representation. Expand

k-GANs: Ensemble of Generative Models with Semi-Discrete Optimal Transport

- Computer Science, Mathematics
- ArXiv
- 2019

A principled method for training an ensemble of GANs using semi-discrete optimal transport theory and the resulting k-GANs algorithm has strong theoretical connection with the k-medoids algorithm. Expand

Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds

- Computer Science, Mathematics
- ArXiv
- 2019

These findings present the first primal-dual relationship between GANs and Autoencoder models, comment on generalization abilities and make a step towards unifying these models. Expand

Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein

- Computer Science, Mathematics
- ICLR
- 2021

This work proposes a new relational discrepancy, named spherical sliced fused Gromov Wasserstein (SSFG), that can find an important area of projections characterized by a von Mises-Fisher distribution and introduces two variants of SSFG to improve its performance. Expand

#### References

SHOWING 1-10 OF 70 REFERENCES

From optimal transport to generative modeling: the VEGAN cookbook

- Mathematics
- 2017

We study unsupervised generative modeling in terms of the optimal transport (OT) problem between true (but unknown) data distribution $P_X$ and the latent variable model distribution $P_G$. We show… Expand

Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model

- Computer Science, Mathematics
- ArXiv
- 2018

Sliced-Wasserstein Autoencoders (SWAE) are introduced, which are generative models that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or defining a closed-form for the distribution. Expand

Learning Generative Models with Sinkhorn Divergences

- Computer Science, Mathematics
- AISTATS
- 2018

This paper presents the first tractable computational method to train large scale generative models using an optimal transport loss, and tackles three issues by relying on two key ideas: entropic smoothing, which turns the original OT loss into one that can be computed using Sinkhorn fixed point iterations; and algorithmic (automatic) differentiation of these iterations. Expand

Wasserstein Auto-Encoders

- Computer Science, Mathematics
- ICLR
- 2018

The Wasserstein Auto-Encoder (WAE) is proposed---a new algorithm for building a generative model of the data distribution that shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality, as measured by the FID score. Expand

Adversarial Autoencoders

- Computer Science
- ArXiv
- 2015

This paper shows how the adversarial autoencoder can be used in applications such as semi-supervised classification, disentangling style and content of images, unsupervised clustering, dimensionality reduction and data visualization, and performed experiments on MNIST, Street View House Numbers and Toronto Face datasets. Expand

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

- Mathematics, Computer Science
- NIPS
- 2017

This work proposes a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions and introduces the "Frechet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. Expand

VAE with a VampPrior

- Computer Science, Mathematics
- AISTATS
- 2018

This paper proposes to extend the variational auto-encoder (VAE) framework with a new type of prior called "Variational Mixture of Posteriors" prior, or VampPrior for short, which consists of a mixture distribution with components given by variational posteriors conditioned on learnable pseudo-inputs. Expand

Hyperspherical Variational Auto-Encoders

- Computer Science, Mathematics
- UAI
- 2018

This work proposes using a von Mises-Fisher distribution instead of a Gaussian distribution for both the prior and posterior of the Variational Auto-Encoder, leading to a hyperspherical latent space. Expand

Fixing a Broken ELBO

- Computer Science, Mathematics
- ICML
- 2018

This framework derives variational lower and upper bounds on the mutual information between the input and the latent variable, and uses these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Expand

Distribution Matching in Variational Inference

- Computer Science, Mathematics
- ArXiv
- 2018

It is concluded that at present, VAE-GAN hybrids have limited applicability: they are harder to scale, evaluate, and use for inference compared to VAEs; and they do not improve over the generation quality of GANs. Expand