• Corpus ID: 246016304

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

@article{Bao2022AnalyticDPMAA,
  title={Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models},
  author={Fan Bao and Chongxuan Li and Jun Zhu and Bo Zhang},
  journal={ArXiv},
  year={2022},
  volume={abs/2201.06503}
}
Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function… 
Fast Sampling of Diffusion Models with Exponential Integrator
TLDR
The proposedDEIS is a fast sampling method for DMs based on the Exponential Integrator designed for discretizing ordinary differential equations (ODEs) and leverages a semilinear structure of the learned diffusion process to reduce the discretization error.
Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models
TLDR
This work considers diagonal and full covariances to improve the expressive power of DPMs, and proposes to estimate the optimal covariance and its correction given imperfect mean by learning conditional expectations.
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
TLDR
This work proposes DPM-Solver, a fast dedicated high-order solver for diffusion ODEs with the convergence order guarantee, suitable for both discrete-time and continuous-time DPMs without any further training.
A Flexible Diffusion Model
Diffusion (score-based) generative models have been widely used for modeling various types of complex data, including images, audios, and point clouds. Recently, the deep connection between
gDDIM: Generalized denoising diffusion implicit models
TLDR
An interpretation of the accelerating effects of DDIM is presented that also explains the advantages of a deterministic sampling scheme over the stochastic one for fast sampling and a small but delicate modification in parameterizing the score network.
Accelerating Score-based Generative Models for High-Resolution Image Synthesis
TLDR
This work introduces a novel Target Distribution Aware Sampling (TDAS) method, which can consistently accelerate state-of-the-art SGMs, particularly on more challenging high resolution image generation tasks by up to 18 .
Elucidating the Design Space of Diffusion-Based Generative Models
TLDR
The theory and practice of diffusion-based generative models are currently unnecessarily convoluted and the design changes dramatically improve both the efficiency and quality obtainable with pre-trained score networks from previous work, including improving the FID of an existing ImageNet-64 model from 2.07 to near-SOTA 1.55.
Few-Shot Diffusion Models
TLDR
Few-Shot Diffusion Models (FSDM), a framework for few-shot generation leveraging conditional DDPMs, and how conditioning the model on patch-based input set information improves training convergence is shown.
Subspace Diffusion Generative Models
TLDR
This framework restricts the diffusion via projections onto subspaces as the data distribution evolves toward noise, which improves sample quality and reduces the computational cost of inference for the same number of denoising steps.
Hierarchical Text-Conditional Image Generation with CLIP Latents
TLDR
This work proposes a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the imageembedding, and shows that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity.
...
...

References

SHOWING 1-10 OF 52 REFERENCES
Variational Diffusion Models
TLDR
A family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often faster optimization.
Learning to Efficiently Sample from Diffusion Probabilistic Models
TLDR
This paper introduces an exact dynamic programming algorithm that finds the optimal discrete time schedules for any pre-trained DDPM, and exploits the fact that ELBO can be decomposed into separate KL terms, and discovers the time schedule that maximizes the training ELBO exactly.
Improved Denoising Diffusion Probabilistic Models
TLDR
This work shows that with a few simple modifications, DDPMs can also achieve competitive log-likelihoods while maintaining high sample quality, and finds that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes with a negligible difference in sample quality.
Denoising Diffusion Probabilistic Models
TLDR
High quality image synthesis results are presented using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics, which naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
TLDR
This work proposes to amplify human effort through a partially automated labeling scheme, leveraging deep learning with humans in the loop, and constructs a new image dataset, LSUN, which contains around one million labeled images for each of 10 scene categories and 20 object categories.
2021), we train 500K iterations with a batch size of 128, use a learning rate
  • 2021
Score-Based Generative Modeling through Stochastic Differential Equations
TLDR
This work presents a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by Slowly removing the noise.
Denoising Diffusion Implicit Models
TLDR
Denoising diffusion implicit models (DDIMs) are presented, a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs that can produce high quality samples faster and perform semantically meaningful image interpolation directly in the latent space.
WaveGrad: Estimating Gradients for Waveform Generation
TLDR
WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality.
Decoupled Weight Decay Regularization
TLDR
This work proposes a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function, and provides empirical evidence that this modification substantially improves Adam's generalization performance.
...
...