Entropy Computations via Analytic Depoissonization

  title={Entropy Computations via Analytic Depoissonization},
  author={Philippe Jacquet and Wojciech Szpankowski},
  journal={IEEE Trans. Inf. Theory},
We investigate the basic question of information theory, namely, evaluation of Shannon entropy, and a more general Renyi (1961) entropy, for some discrete distributions (e.g., binomial, negative binomial, etc.). We aim at establishing analytic methods (i.e., those in which complex analysis plays a pivotal role) for such computations which often yield estimates of unparalleled precision. The main analytic tool used here is that of analytic poissonization and depoissonization. We illustrate our… 

Figures from this paper

Entropy computations for discrete distributions: towards analytic information theory

  • P. JacquetW. Szpankowski
  • Mathematics, Computer Science
    Proceedings. 1998 IEEE International Symposium on Information Theory (Cat. No.98CH36252)
  • 1998
This work investigates the basic question of information theory, namely, evaluation of Shannon entropy, and a more general Renyi entropy, for some discrete distributions, and aims at establishing analytic methods for such computations which often yield estimates of unparalleled precision.

Asymptotic average redundancy of Huffman (and other) block codes

The appearance of the fractal-like function explains the erratic behavior of the Huffman redundancy, and its "resistance" to succumb to a precise analysis.

Relative Fisher information of discrete classical orthogonal polynomials

The analytic information theory of discrete distributions was initiated in 1998 by C. Knessl, P. Jacquet and S. Szpankowski who addressed the precise evaluation of the Renyi and Shannon entropies of

Sharp Bounds on the Entropy of the Poisson Law and Related Quantities

Upper and lower bounds for H(¿) are derived that are asymptotically tight and easy to compute and follow easily on the relative entropy D(n, p) between a binomial and a Poisson.

Analytic variations on redundancy rates of renewal processes

A precise estimate of the redundancy rate for such (nonstationary) renewal sources, namely, 2/(log2)/spl radic/((/spl pi//sup 2//6-1)n)+O(log n) is given, derived by complex-analytic methods that include generating function representations, Mellin transforms, singularity analysis, and saddle-point estimates.

The Depoissonisation Quintet: Rice-Poisson-Mellin-Newton-Laplace

It is proved that the Rice-Mellin path is both of easy and practical use : even though (much?) less general than the Depoissonisation path, it is easier to apply.


It is shown that $\lim _{n\to \infty }\log [n-b(n,1/2)]/\log n =\theta =\log [(1+\sqrt {5})/2]$ (where $\log$ is the logarithm to base $2$), which together with limited numerical results suggests that n-b (n, 1/2) may be a regularly varying sequence of index $\theta$.

Expected External Profile of PATRICIA Tries

This work considers PATRICIA tries on n random binary strings generated by a memoryless source with parameter p ≥ 1/2 and analyzes asymptotic matching and linearization of the expected value of the external profile at level k = k(n), defined to be the number of leaves atlevel k.

On Delta-Method of Moments and Probabilistic Sums

A general framework for determining asymptotics of the expected value of random variables of the form f(X) in terms of a function f and central moments of the random variable X and its specific extension to random variables which are sums of identically distributed independent random variables is discussed.

Maximum Likelihood Estimation of Functionals of Discrete Distributions

The worst case squared error risk incurred by the maximum likelihood estimator (MLE) in estimating the Shannon entropy is described and it is established that the MLE achieves the minimax optimal rate regardless of the alphabet size.



Asymptotic redundancies for universal quantum coding

This work compares the quantum asymptotic redundancy formulas derived by naively applying the (nonquantum) counterparts of Clarke and Barren, and finds certain common features.

Average redundancy rate of the Lempel-Ziv code

It is proved that for a memoryless source the average redundancy rate attains asymptotically Er/sub n/=(A+/spl delta/(n))/log n+O(loglog n/log/sup 2/n) where A is an explicitly given constant that depends on the source characteristics, and /spl delta)/x is a fluctuating function.

Analysis of digital tries with Markovian dependency

A complete characterization of a digital tree is presented from the depth viewpoint in a Markovian framework, that is, under the assumption that symbols in a key are Markov-dependent, and shows that D/sub n/ tends to the normal distribution in all cases except the symmetric independent model.

Techniques of the Average Case Analysis of Algorithms

This is an extended version of a book chapter that I wrote for the Handbook on Algorithms and Theon) of Computation (Ed. M. Atallah) in which some probabilistic and analytical techniques of the


This paper provides asymptotic expansions of certain recurrences studied there which describe the optimal redundancy of universal codes which was recently investigated by Shtarkov et al.

Asymptotic Behavior of the Lempel-Ziv Parsing Scheme and Digital Search Trees

Average Profile of the Generalized Digital Search Tree and the Generalized Lempel-Ziv Algorithm

The depth of a randomly selected node in a generalized digital search tree and the length of arandomly selected phrase in the generalized Lempel--Ziv parsing scheme are investigated.

Average profile and limiting distribution for a phrase size in the Lempel-Ziv parsing algorithm

The size of a randomly selected phrase, and the average number of phrases of a given size (the so-called average profile of phrase sizes) are focused on.

Handbook of Combinatorics

Part 1 Structures: graphs - basic graph theory - paths and circuits, J.A. Bondy, connectivity and network flows, A. Frank, matchings and extensions, W.R. Pulleyblank, colouring, stable sets and