Corpus ID: 211010797

Generative Modeling with Denoising Auto-Encoders and Langevin Sampling

  title={Generative Modeling with Denoising Auto-Encoders and Langevin Sampling},
  author={Adam Block and Youssef Mroueh and Alexander Rakhlin},
We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes. We overcome the challenge of relying only on $L^2$ bounds on the score estimation error and provide… Expand
Fast Mixing of Multi-Scale Langevin Dynamics under the Manifold Hypothesis
This work demonstrates how the manifold hypothesis allows for the considerable reduction of mixing time, from exponential in the ambient dimension to depending only on the (much smaller) intrinsic dimension of the data. Expand
Joint Intensity-Gradient Guided Generative Modeling for Colorization
The joint intensity-gradient constraint in data-fidelity term is proposed to limit the degree of freedom within generative model at the iterative colorization stage, and it is conducive to edge-preserving. Expand
Iterative Reconstruction for Low-Dose CT using Deep Gradient Priors of Generative Model
This work integrates the data-consistency as a conditional term into the iterative generative model for low-dose CT, and uses the annealing Langevin dynamics to update the trained priors with conditional scheme. Expand
Dimension-free log-Sobolev inequalities for mixture distributions
We prove that if (Px)x∈X is a family of probability measures which satisfy the log-Sobolev inequality and whose pairwise chi-squared divergences are uniformly bounded, and μ is any mixingExpand
Deep Generative Learning via Schrödinger Bridge
The theoretical results guarantee that the distribution learned by the approach converges to the target distribution and indicate that the generative model via Schrödinger Bridge is comparable with state-ofthe-art GANs, suggesting a new formulation of generative learning. Expand
High-dimensional Assisted Generative Model for Color Image Restoration
  • Kai Hong, Chunhua Wu, +4 authors Qiegen Liu
  • Computer Science, Engineering
  • ArXiv
  • 2021
The proposed unsupervised learning and iterative restoration algorithm, which involves a pre-trained generative network to obtain prior, has transparent and clear interpretation compared to other data-driven approaches. Expand
Homotopic Gradients of Generative Density Priors for MR Image Reconstruction
This work trains a more powerful noise conditional score network by forming high-dimensional tensor as the network input at the training phase and estimates the target gradients in higher-dimensional space to tackle low-dimensional manifold and low data density region issues in generative density prior. Expand
Inference and De-Noising of Non-Gaussian Particle Distribution Functions: A Generative Modeling Approach
The use of normalizing flows is demonstrated to learn a smooth, tractable approximation to the noisy particle distribution function and it is demonstrated that the resulting data driven likelihood conserves relevant physics and may be extended to encapsulate the temporal evolution of the distribution function. Expand
On Energy-Based Models with Overparametrized Shallow Neural Networks
This work shows that models trained in the so-called ’active’ regime provide a statistical advantage over their associated ’lazy’ or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Expand


Generative Modeling by Estimating Gradients of the Data Distribution
A new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching, which allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Expand
What regularized auto-encoders learn from the data-generating distribution
It is shown that the auto-encoder captures the score (derivative of the log-density with respect to the input) and contradicts previous interpretations of reconstruction error as an energy function. Expand
Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
The present work provides a nonasymptotic analysis in the context of non-convex learning problems, giving finite-time guarantees for SGLD to find approximate minimizers of both empirical and population risks. Expand
A Connection Between Score Matching and Denoising Autoencoders
A proper probabilistic model for the denoising autoencoder technique is defined, which makes it in principle possible to sample from them or rank examples by their energy, and a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives is suggested. Expand
Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond
The convergence rate of sampling algorithms obtained by discretizing smooth Ito diffusions exhibiting fast Wasserstein-$2$ contraction is established, based on local deviation properties of the integration scheme. Expand
Generative Adversarial Nets
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and aExpand
Estimation of Wasserstein distances in the Spiked Transport Model
We propose a new statistical model, the spiked transport model, which formalizes the assumption that two probability distributions differ only on a low-dimensional subspace. We study the minimax rateExpand
Sampling from a Log-Concave Distribution with Projected Langevin Monte Carlo
We extend the Langevin Monte Carlo (LMC) algorithm to compactly supported measures via a projection step, akin to projected stochastic gradient descent (SGD). We show that (projected) LMC allows toExpand
Sparse Regression Learning by Aggregation and Langevin Monte-Carlo
A sharp PAC-Bayesian type bound for the exponentially weighted aggregate (EWA) under the expected squared empirical loss is proved and it is valid even for unbounded regression functions and the choice of the temperature parameter depends exclusively on the noise level. Expand
Theoretical guarantees for approximate sampling from smooth and log‐concave densities
Sampling from various kinds of distributions is an issue of paramount importance in statistics since it is often the key ingredient for constructing estimators, test procedures or confidenceExpand