Entropy estimation of symbol sequences.

@article{Schurmann1996EntropyEO,
  title={Entropy estimation of symbol sequences.},
  author={Thomas Schurmann and Peter Grassberger},
  journal={Chaos},
  year={1996},
  volume={6 3},
  pages={
          414-427
        }
}
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limits for the space and time complexities of the compression algorithms. A scaling law is proposed for extrapolation from finite sample lengths. This is applied to sequences of… 

Scaling behaviour of entropy estimates

The scaling behaviour of certain non-probabilistic entropy estimation schemes suggested by Lempel and Ziv in the context of algorithmic complexity and data compression are investigated, with the scaling variable being the length N of the sequence.

A Note on the Shannon Entropy of Short Sequences

For source sequences of length L symbols, the idea is to use a more realistic value to the usual benchmark of number of code letters by source letters, F(U), which corresponds to the second central moment of the random variable that measures the information content of a source symbol.

Entropy Gradient : A Technique for Estimating The Entropy of Finite Time Series

A new technique is presented using varying levels of approximations for the entropy of a finite sequence of symbols taken from a fini te alphabet to result in an estimate to the absolute entropy.

Effective normalization of complexity measurements for epoch length and sampling frequency.

It is demonstrated that algorithmic redundancy is not sensitive to message length or to observation scale (sampling frequency) when stationary systems are examined.

Computing entropy rate of symbol sources & a distribution-free limit theorem

A fundamentally new approach to estimate entropy rates is presented, which is demonstrated to converge significantly faster in terms of input data lengths, and is shown to be effective in diverse applications ranging from the estimation of the entropy rate of English texts toThe estimation of complexity of chaotic dynamical systems.

Determining the Number of Samples Required to Estimate Entropy in Natural Sequences

A method is proposed for estimating the number of samples required to compute Shannon entropy for a set of ranked symbolic “natural” events using a modified Zipf-Mandelbrot law and the Dvoretzky–Kiefer–Wolfowitz inequality.

Batch and stream entropy with fixed partitions for chaos-based random bit generators.

Measures are proposed for reliably estimating the entropy of bits produced in an entropy source using a chaotic physical system and depend on the end-to-end method of transfer of entropy from the chaoticphysical system to the bit entropy source.

Selecting an Effective Entropy Estimator for Short Sequences of Bits and Bytes with Maximum Entropy

In this work, the most effective estimator is selected to estimate entropy in short samples of bytes and bits with maximum entropy, based on its bias, the mean square error long samples ofbytes and bits.

Large Deviations for Empirical Entropies of Gibbsian sources

The entropy of an ergodic finite-alphabet process can be computed from a single typical sample path x1 using the entropy of the k-block empirical probability and letting k grow with n roughly like

Regularities unseen, randomness observed: levels of entropy convergence.

Several phenomenological approaches to applying information theoretic measures of randomness and memory to stochastic and deterministic processes are synthesized by using successive derivatives of the Shannon entropy growth curve to look at the relationships between a process's entropy convergence behavior and its underlying computational structure.
...

References

SHOWING 1-10 OF 56 REFERENCES

Estimating the information content of symbol sequences and efficient codes

Several variants of an algorithm for estimating Shannon entropies of symbol sequences are presented, which seem to be the optimal algorithms for sequences with strong long-range correlations, e.g. natural languages.

A New Method to Calculate Higher-Order Entropies from Finite Samples

The finite size of natural symbol sequences, e.g. DNA strands or texts in natural languages, is responsible for the worsening statistics with respect to increasing length n of the substrings, thus

Compression of individual sequences via variable-rate coding

The proposed concept of compressibility is shown to play a role analogous to that of entropy in classical information theory where one deals with probabilistic ensembles of sequences rather than with individual sequences.

Entropy and Long-Range Correlations in Literary English

We investigated long-range correlations in two literary texts, "Moby Dyck" by H. Melville and Grimm's tales. The analysis is based on the calculation of entropylike quantities as the mutual

Entropy of symbolic sequences: the role of correlations

It is shown that sporadic systems give rise to peculiar scaling properties as a result of long-range correlations, and the potential implications of this possibility in the structure of natural languages are explored.

Enlarged scaling ranges for the KS-entropy and the information dimension.

Rewriting differences of block entropies as averages over decay rates, and ignoring all parts of the sample where these rates are uncomputable because of the lack of neighbours, yields improved entropy estimates, and the scaling range for estimates of the information dimension can be extended considerably.

On the Complexity of Finite Sequences

A new approach to the problem of evaluating the complexity ("randomness") of finite sequences is presented, related to the number of steps in a self-delimiting production process by which a given sequence is presumed to be generated.

A universal data compression system

A universal data compression algorithm is described which is capable of compressing long strings generated by a "finitely generated" source, with a near optimum per symbol length without prior
...