Corpus ID: 8528836

Minimax Estimation of KL Divergence between Discrete Distributions

@article{Han2016MinimaxEO,
  title={Minimax Estimation of KL Divergence between Discrete Distributions},
  author={Yanjun Han and Jiantao Jiao and Tsachy Weissman},
  journal={ArXiv},
  year={2016},
  volume={abs/1605.09124}
}
  • Yanjun Han, Jiantao Jiao, Tsachy Weissman
  • Published 2016
  • Computer Science, Mathematics
  • ArXiv
  • We consider the problem of estimating the KL divergence between two discrete probability measures $P$ and $Q$ from empirical data in a non-asymptotic and possibly large alphabet setting. We construct minimax rate-optimal estimators for $D(P\|Q)$ when the likelihood ratio is upper bounded by a constant which may depend on the support size, and show that the performance of the optimal estimator with $n$ samples is essentially that of the Maximum Likelihood Estimator (MLE) with $n\ln n$ samples… CONTINUE READING

    Figures and Topics from this paper.

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 16 CITATIONS

    Minimax rate-optimal estimation of KL divergence between discrete distributions

    Maximum Likelihood Estimation of Functionals of Discrete Distributions

    VIEW 12 EXCERPTS
    CITES BACKGROUND

    Minimax Estimation of Functionals of Discrete Distributions

    Profile Maximum Likelihood is Optimal for Estimating KL Divergence

    • Jayadev Acharya
    • Computer Science
    • 2018 IEEE International Symposium on Information Theory (ISIT)
    • 2018
    VIEW 2 EXCERPTS
    CITES BACKGROUND

    Beyond maximum likelihood: Boosting the Chow-Liu algorithm for large alphabets

    Estimation of KL Divergence: Optimal Minimax Rate

    VIEW 6 EXCERPTS
    CITES METHODS, BACKGROUND & RESULTS
    HIGHLY INFLUENCED

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 42 REFERENCES

    Minimax Estimation of Functionals of Discrete Distributions

    VIEW 9 EXCERPTS

    Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation

    VIEW 3 EXCERPTS
    HIGHLY INFLUENTIAL

    Kullback-Leibler divergence estimation of continuous distributions

    • Fernando Pérez-Cruz
    • Computer Science, Mathematics
    • 2008 IEEE International Symposium on Information Theory
    • 2008

    The Power of Linear Estimators

    VIEW 4 EXCERPTS
    HIGHLY INFLUENTIAL

    Nonparametric Estimation of Küllback-Leibler Divergence

    VIEW 1 EXCERPT

    Estimating the Unseen