# Normalizing Flows for Probabilistic Modeling and Inference

@article{Papamakarios2021NormalizingFF, title={Normalizing Flows for Probabilistic Modeling and Inference}, author={George Papamakarios and Eric T. Nalisnick and Danilo Jimenez Rezende and Shakir Mohamed and Balaji Lakshminarayanan}, journal={J. Mach. Learn. Res.}, year={2021}, volume={22}, pages={57:1-57:64} }

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows…

## 578 Citations

### Graphical Normalizing Flows

- Computer ScienceAISTATS
- 2021

The graphical normalizing flow is proposed, a new invertible transformation with either a prescribed or a learnable graphical structure that provides a promising way to inject domain knowledge into normalizing flows while preserving both the interpretability of Bayesian networks and the representation capacity ofnormalizing flows.

### Principled Interpolation in Normalizing Flows

- Computer ScienceECML/PKDD
- 2021

This paper uses the Dirichlet and von Mises-Fisher base distributions to enforce a fixed norm and change the base distribution, to allow for a principled way of interpolation, and shows superior performance in terms of bits per dimension, Fr\'echet Inception Distance (FID), and Kernel Inception distance (KID) scores.

### Mixture of Discrete Normalizing Flows for Variational Inference

- Computer ScienceArXiv
- 2020

This work presents a novel algorithm for modeling the posterior distribution of models with discrete latent variables, based on boosting variational inference, and considers mixtures of discrete normalizing flows instead.

### Automatic variational inference with cascading flows

- Computer ScienceICML
- 2021

Cascading flows are introduced, a new family of variational programs that can be constructed automatically from an input probabilistic program and can also be amortized automatically that have much higher performance than both normalizing flows and ASVI in a large set of structured inference problems.

### Transforming Gaussian Processes With Normalizing Flows

- Computer ScienceAISTATS
- 2021

A variational approximation to the resulting Bayesian inference problem is derived, which is as fast as stochastic variational GP regression and makes the model a computationally efficient alternative to other hierarchical extensions of GP priors.

### Stochastic Normalizing Flows

- Computer Science, MathematicsNeurIPS
- 2020

Stochastic Normalizing Flows (SNF) is proposed -- an arbitrary sequence of deterministic invertible functions and stochastic sampling blocks that illustrate the representational power, sampling efficiency and asymptotic correctness of SNFs on several benchmarks including applications to sampling molecular systems in equilibrium.

### Approximate Probabilistic Inference with Composed Flows

- Computer Science
- 2020

This work proposes a framework for probabilistic inference that trains a new generative model with the property that its composition with the given model approximates the target conditional distribution and can efficiently train it using variational inference and also handle conditioning under arbitrary differentiable transformations.

### Stochastic Normalizing Flows

- Computer Science, Mathematics
- 2020

Stochastic Normalizing Flows (SNF) is proposed – an arbitrary sequence of deterministic invertible functions and stochastic sampling blocks and illustrated the representational power, sampling efficiency and asymptotic correctness of SNFs on several benchmarks including applications to sampling molecular systems in equilibrium.

### Normalizing Flows: An Introduction and Review of Current Methods

- BusinessIEEE Transactions on Pattern Analysis and Machine Intelligence
- 2021

The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning to provide context and explanation of the models.

### Flexible Approximate Inference via Stratified Normalizing Flows

- Mathematics, Computer ScienceUAI
- 2020

An approximate inference procedure is developed that allows explicit control of the bias/variance tradeoff, interpolating between the sampling and the variational regime, and uses a normalizing flow to map the integrand onto a uniform distribution.

## References

SHOWING 1-10 OF 150 REFERENCES

### Variational Inference with Normalizing Flows

- Computer Science, MathematicsICML
- 2015

It is demonstrated that the theoretical advantages of having posteriors that better match the true posterior, combined with the scalability of amortized variational approaches, provides a clear improvement in performance and applicability of variational inference.

### Latent Normalizing Flows for Discrete Sequences

- Computer ScienceICML
- 2019

A VAE-based generative model is proposed which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space in this setting, finding that it is crucial for the flow- based distribution to be highly multimodal.

### Normalizing Flows: Introduction and Ideas

- Computer ScienceArXiv
- 2019

A Normalizing Flow (NF) is family of generative models which produces tractable distributions where both sampling and density evaluation can be efficient and exact.

### Localised Generative Flows

- MathematicsArXiv
- 2019

It is proved that a flow must become arbitrarily numerically noninvertible in order to approximate the target closely, and proposed Continuously Indexed Flows (CIFs) are proposed, which replace the single bijection used by normalising flows with a continuously indexed family of bijections.

### Residual Flows for Invertible Generative Modeling

- MathematicsNeurIPS
- 2019

The resulting approach, called Residual Flows, achieves state-of-the-art performance on density estimation amongst flow-based models, and outperforms networks that use coupling blocks at joint generative and discriminative modeling.

### Normalizing Flows: An Introduction and Review of Current Methods

- BusinessIEEE Transactions on Pattern Analysis and Machine Intelligence
- 2021

The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning to provide context and explanation of the models.

### Auto-Encoding Variational Bayes

- Computer ScienceICLR
- 2014

A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.

### Learning in Implicit Generative Models

- Computer ScienceArXiv
- 2016

This work develops likelihood-free inference methods and highlight hypothesis testing as a principle for learning in implicit generative models, using which it is able to derive the objective function used by GANs, and many other related objectives.

### Discrete Flows: Invertible Generative Models of Discrete Data

- Computer Science, MathematicsDGS@ICLR
- 2019

It is shown that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations.

### Sylvester Normalizing Flows for Variational Inference

- MathematicsUAI
- 2018

Sylvester normalizing flows remove the well-known single-unit bottleneck from planar flows, making a single transformation much more flexible, and are compared against planarflows and inverse autoregressive flows.