Corpus ID: 8528836

# Minimax Estimation of KL Divergence between Discrete Distributions

@article{Han2016MinimaxEO,
title={Minimax Estimation of KL Divergence between Discrete Distributions},
author={Yanjun Han and Jiantao Jiao and Tsachy Weissman},
journal={ArXiv},
year={2016},
volume={abs/1605.09124}
}
• Published 2016
• Computer Science, Mathematics
• ArXiv
• We consider the problem of estimating the KL divergence between two discrete probability measures $P$ and $Q$ from empirical data in a non-asymptotic and possibly large alphabet setting. We construct minimax rate-optimal estimators for $D(P\|Q)$ when the likelihood ratio is upper bounded by a constant which may depend on the support size, and show that the performance of the optimal estimator with $n$ samples is essentially that of the Maximum Likelihood Estimator (MLE) with $n\ln n$ samples… CONTINUE READING

#### Citations

##### Publications citing this paper.
SHOWING 1-10 OF 16 CITATIONS

## Minimax rate-optimal estimation of KL divergence between discrete distributions

• Mathematics, Medicine, Computer Science
• 2016 International Symposium on Information Theory and Its Applications (ISITA)
• 2016

## Maximum Likelihood Estimation of Functionals of Discrete Distributions

• Mathematics, Computer Science
• IEEE Transactions on Information Theory
• 2017
VIEW 12 EXCERPTS
CITES BACKGROUND

## Minimax Estimation of Functionals of Discrete Distributions

• Computer Science, Medicine, Mathematics
• IEEE Transactions on Information Theory
• 2015

## The Bethe and Sinkhorn Permanents of Low Rank Matrices and Implications for Profile Maximum Likelihood

• Mathematics, Computer Science
• ArXiv
• 2020

## Efficient profile maximum likelihood for universal symmetric property estimation

• Computer Science, Mathematics
• STOC
• 2019
VIEW 1 EXCERPT
CITES BACKGROUND

## Profile Maximum Likelihood is Optimal for Estimating KL Divergence

• Computer Science
• 2018 IEEE International Symposium on Information Theory (ISIT)
• 2018
VIEW 2 EXCERPTS
CITES BACKGROUND

## Beyond maximum likelihood: Boosting the Chow-Liu algorithm for large alphabets

• Computer Science
• 2016 50th Asilomar Conference on Signals, Systems and Computers
• 2016

## Minimax Rates of Estimating Approximate Differential Privacy

• Computer Science, Mathematics
• NeurIPS 2019
• 2019

## Local moment matching: A unified methodology for symmetric functional estimation and distribution estimation under Wasserstein distance

• Computer Science, Mathematics
• COLT
• 2018
VIEW 6 EXCERPTS
CITES METHODS & BACKGROUND

## Estimation of KL Divergence: Optimal Minimax Rate

• Computer Science, Mathematics
• IEEE Transactions on Information Theory
• 2018
VIEW 6 EXCERPTS
CITES METHODS, BACKGROUND & RESULTS
HIGHLY INFLUENCED

#### References

##### Publications referenced by this paper.
SHOWING 1-10 OF 42 REFERENCES

## Minimax Estimation of Functionals of Discrete Distributions

• Computer Science, Medicine, Mathematics
• IEEE Transactions on Information Theory
• 2015
VIEW 9 EXCERPTS

## Chebyshev polynomials, moment matching, and optimal estimation of the unseen

• Mathematics
• 2015
VIEW 1 EXCERPT

## The Complexity of Estimating R\'enyi Entropy

• Mathematics
• 2014

## Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization

• Computer Science, Mathematics
• IEEE Transactions on Information Theory
• 2010
VIEW 1 EXCERPT

## Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation

• Computer Science, Mathematics
• IEEE Transactions on Information Theory
• 2016
VIEW 3 EXCERPTS
HIGHLY INFLUENTIAL

## Kullback-Leibler divergence estimation of continuous distributions

• Fernando Pérez-Cruz
• Computer Science, Mathematics
• 2008 IEEE International Symposium on Information Theory
• 2008

## The Power of Linear Estimators

• Mathematics, Computer Science
• 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science
• 2011
VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL

## Nonparametric Estimation of Küllback-Leibler Divergence

• Medicine, Mathematics, Computer Science
• Neural Computation
• 2014
VIEW 1 EXCERPT

## Estimating the Unseen

• Computer Science, Mathematics
• NIPS
• 2013

## TESTING COMPOSITE HYPOTHESES, HERMITE POLYNOMIALS AND OPTIMAL ESTIMATION OF A NONSMOOTH FUNCTIONAL

• Mathematics
• 2011