• Corpus ID: 536962

Generative Moment Matching Networks

@article{Li2015GenerativeMM,
  title={Generative Moment Matching Networks},
  author={Yujia Li and Kevin Swersky and Richard S. Zemel},
  journal={ArXiv},
  year={2015},
  volume={abs/1502.02761}
}
We consider the problem of learning deep generative models from data. We formulate a method that generates an independent sample via a single feedforward pass through a multilayer perceptron, as in the recently proposed generative adversarial networks (Goodfellow et al., 2014). Training a generative adversarial network, however, requires careful optimization of a difficult minimax program. Instead, we utilize a technique from statistical hypothesis testing known as maximum mean discrepancy (MMD… 
View PDF on arXiv
View With Semantic Reader
668 Citations
Highly Influential Citations
100
Background Citations
328
Methods Citations
287
Results Citations
13

Figures and Tables from this paper

668 Citations

An Online Learning Approach to Generative Adversarial Networks
TLDR
A novel training method named Chekhov GAN is proposed and it is shown that this method provably converges to an equilibrium for semi-shallow GAN architectures, i.e. architectures where the discriminator is a one layer network and the generator is arbitrary.
Training generative neural networks via Maximum Mean Discrepancy optimization
TLDR
This work considers training a deep neural network to generate samples from an unknown distribution given i.i.d. data to frame learning as an optimization minimizing a two-sample test statistic, and proves bounds on the generalization error incurred by optimizing the empirical MMD.
Generative Ratio Matching Networks
TLDR
This work takes the insight of using kernels as fixed adversaries further and presents a novel method for training deep generative models that does not involve saddlepoint optimization, called generative ratio matching or GRAM for short.
Online Kernel based Generative Adversarial Networks
TLDR
It is shown empirically that OKGANs empirically perform dramatically better, with respect to reverse KL-divergence, than other GAN formulations on synthetic data; on classical vision datasets such as MNIST, SVHN, and CelebA, show comparable performance.
NONPARAMETRIC APPROACHES FOR TRAINING DEEP GENERATIVE NETWORKS
TLDR
It is conjecture that the nonparametric approach for training DNNs can provide a viable alternative to the popular GAN formulations and a new algorithm based on the Prokhorov metric between distributions is developed, which is believed to provide promising results on certain kinds of data.
Adaptive Density Estimation for Generative Models
TLDR
This work shows that their model significantly improves over existing hybrid models: offering GAN-like samples, IS and FID scores that are competitive with fully adversarial models and improved likelihood scores.
On the Quantitative Analysis of Decoder-Based Generative Models
TLDR
This work proposes to use Annealed Importance Sampling for evaluating log-likelihoods for decoder-based models and validate its accuracy using bidirectional Monte Carlo, and analyzes the performance of decoded models, the effectiveness of existing log- likelihood estimators, the degree of overfitting, and the degree to which these models miss important modes of the data distribution.
Improved Techniques for Training GANs
TLDR
This work focuses on two applications of GANs: semi-supervised learning, and the generation of images that humans find visually realistic, and presents ImageNet samples with unprecedented resolution and shows that the methods enable the model to learn recognizable features of ImageNet classes.
Adversarial Autoencoders
TLDR
This paper shows how the adversarial autoencoder can be used in applications such as semi-supervised classification, disentangling style and content of images, unsupervised clustering, dimensionality reduction and data visualization, and performed experiments on MNIST, Street View House Numbers and Toronto Face datasets.
Generative mixture of networks
TLDR
A generative model based on training deep architectures that consists of K networks that are trained together to learn the underlying distribution of a given data set, called Mixture of Networks, has high capability in characterizing complicated data distributions as well as clustering data.
...
...

References

SHOWING 1-10 OF 53 REFERENCES
Generative Adversarial Nets
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a…
Training generative neural networks via Maximum Mean Discrepancy optimization
TLDR
This work considers training a deep neural network to generate samples from an unknown distribution given i.i.d. data to frame learning as an optimization minimizing a two-sample test statistic, and proves bounds on the generalization error incurred by optimizing the empirical MMD.
Deep Generative Stochastic Networks Trainable by Backprop
TLDR
Theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders are provided and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood.
A Generative Process for sampling Contractive Auto-Encoders
TLDR
A procedure for generating samples that are consistent with the local structure captured by a contractive auto-encoder and which experimentally appears to converge quickly and mix well between modes, compared to Restricted Boltzmann Machines and Deep Belief Networks is proposed.
A Winner-Take-All Method for Training Sparse Convolutional Autoencoders
TLDR
A way to train convolutio nal autoencoders layer by layer, where in each layer sparsity is achieved usin g a winner-take-all activation function within each feature map.
Extracting and composing robust features with denoising autoencoders
TLDR
This work introduces and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern.
Generalized Denoising Auto-Encoders as Generative Models
TLDR
A different attack on the problem is proposed, which deals with arbitrary (but noisy enough) corruption, arbitrary reconstruction loss, handling both discrete and continuous-valued variables, and removing the bias due to non-infinitesimal corruption noise.
Greedy Layer-Wise Training of Deep Networks
TLDR
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
Show and tell: A neural image caption generator
TLDR
This paper presents a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image.
Contractive Auto-Encoders: Explicit Invariance During Feature Extraction
TLDR
It is found empirically that this penalty helps to carve a representation that better captures the local directions of variation dictated by the data, corresponding to a lower-dimensional non-linear manifold, while being more invariant to the vast majority of directions orthogonal to the manifold.
...
...