# Featurized Density Ratio Estimation

@inproceedings{Choi2021FeaturizedDR, title={Featurized Density Ratio Estimation}, author={Kristy Choi and Madeline Liao and Stefano Ermon}, booktitle={UAI}, year={2021} }

Density ratio estimation serves as an important technique in the unsupervised machine learning toolbox. However, such ratios are difﬁcult to estimate for complex, high-dimensional data, particu-larly when the densities of interest are sufﬁciently different. In our work, we propose to leverage an invertible generative model to map the two distributions into a common feature space prior to estimation. This featurization brings the densities closer together in latent space, sidestepping…

## Figures and Tables from this paper

## 4 Citations

Density Ratio Estimation via Infinitesimal Classification

- Computer ScienceAISTATS
- 2022

This work proposes DRE- 8, a divide-and-conquer approach to reduce DRE to a series of easier subproblems, and shows that traditional (Stein) scores can be used to obtain integration paths that connect regions of high density in both distributions, improving performance in practice.

A Unified Framework for Multi-distribution Density Ratio Estimation

- Computer Science, MathematicsArXiv
- 2021

A general framework from the perspective of Bregman divergence minimization is developed, justifying the use of any strictly proper scoring rule composite with a link function for multi-distribution DRE and leading to methods that strictly generalize their counterparts in binary DRE, as well as new methods that show comparable or superior performance on various downstream tasks.

Unified Perspective on Probability Divergence via Maximum Likelihood Density Ratio Estimation: Bridging KL-Divergence and Integral Probability Metrics

- Computer ScienceArXiv
- 2022

It is shown that the KL-divergence and the IPMs can be represented as maximal likelihoods differing only by sampling schemes, and this result is used to derive a unified form ofThe IPMs and a relaxed estimation method.

Approximate Data Deletion in Generative Models

- Computer ScienceArXiv
- 2022

This paper proposes a density-ratio-based framework for generative models and introduces a fast method for approximate data deletion and a statistical test for estimating whether or not training points have been deleted.

## References

SHOWING 1-10 OF 63 REFERENCES

Telescoping Density-Ratio Estimation

- Computer ScienceNeurIPS
- 2020

This work introduces a new framework, telescoping density-ratio estimation (TRE), that enables the estimation of ratios between highly dissimilar densities in high-dimensional spaces and demonstrates that TRE can yield substantial improvements over existing single-Ratio methods for mutual information estimation, representation learning and energy-based modelling.

Direct Density Ratio Estimation for Large-scale Covariate Shift Adaptation

- Computer ScienceSDM
- 2008

This work proposes a novel method that allows us to directly estimate the importance from samples without going through the hard task of density estimation, and demonstrates that the proposed method is computationally more efficient than existing approaches with comparable accuracy.

MADE: Masked Autoencoder for Distribution Estimation

- Computer ScienceICML
- 2015

This work introduces a simple modification for autoencoder neural networks that yields powerful generative models and proves that this approach is competitive with state-of-the-art tractable distribution estimators.

A RAD approach to deep mixture models

- Computer ScienceDGS@ICLR
- 2019

This Real and Discrete (RAD) approach retains the desirable normalizing flow properties of exact sampling, exact inference, and analytically computable probabilities, while at the same time allowing simultaneous modeling of both continuous and discrete structure in a data distribution.

Density estimation using Real NVP

- Computer ScienceICLR
- 2017

This work extends the space of probabilistic models using real-valued non-volume preserving (real NVP) transformations, a set of powerful invertible and learnable transformations, resulting in an unsupervised learning algorithm with exact log-likelihood computation, exact sampling, exact inference of latent variables, and an interpretable latent space.

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

- Computer Science, MathematicsICLR
- 2019

This paper uses Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density and demonstrates the approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling.

Learning Likelihoods with Conditional Normalizing Flows

- Computer ScienceArXiv
- 2019

This work provides an effective method to train continuous CNFs for binary problems and applies them to super-resolution and vessel segmentation tasks demonstrating competitive performance on standard benchmark datasets in terms of likelihood and conventional metrics.

Invertible Residual Networks

- Computer ScienceICML
- 2019

The empirical evaluation shows that invertible ResNets perform competitively with both state-of-the-art image classifiers and flow-based generative models, something that has not been previously achieved with a single architecture.

Relative Density-Ratio Estimation for Robust Distribution Comparison

- Computer ScienceNeural Computation
- 2013

This letter uses relative divergences for distribution comparison, which involves approximation of relative density ratios, and shows that the proposed divergence estimator has asymptotic variance independent of the model complexity under a parametric setup, implying that the suggested estimator hardly overfits even with complex models.

Glow: Generative Flow with Invertible 1x1 Convolutions

- Computer ScienceNeurIPS
- 2018

Glow, a simple type of generative flow using an invertible 1x1 convolution, is proposed, demonstrating that a generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images.