# Entropy estimation of symbol sequences.

@article{Schurmann1996EntropyEO, title={Entropy estimation of symbol sequences.}, author={Thomas Schurmann and Peter Grassberger}, journal={Chaos}, year={1996}, volume={6 3}, pages={ 414-427 } }

We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limits for the space and time complexities of the compression algorithms. A scaling law is proposed for extrapolation from finite sample lengths. This is applied to sequences of…

## Figures and Tables from this paper

## 276 Citations

### Scaling behaviour of entropy estimates

- Computer Science
- 2002

The scaling behaviour of certain non-probabilistic entropy estimation schemes suggested by Lempel and Ziv in the context of algorithmic complexity and data compression are investigated, with the scaling variable being the length N of the sequence.

### A Note on the Shannon Entropy of Short Sequences

- Computer ScienceArXiv
- 2018

For source sequences of length L symbols, the idea is to use a more realistic value to the usual benchmark of number of code letters by source letters, F(U), which corresponds to the second central moment of the random variable that measures the information content of a source symbol.

### Effective normalization of complexity measurements for epoch length and sampling frequency.

- Computer SciencePhysical review. E, Statistical, nonlinear, and soft matter physics
- 2001

It is demonstrated that algorithmic redundancy is not sensitive to message length or to observation scale (sampling frequency) when stationary systems are examined.

### Computing entropy rate of symbol sources & a distribution-free limit theorem

- Computer Science2014 48th Annual Conference on Information Sciences and Systems (CISS)
- 2014

A fundamentally new approach to estimate entropy rates is presented, which is demonstrated to converge significantly faster in terms of input data lengths, and is shown to be effective in diverse applications ranging from the estimation of the entropy rate of English texts toThe estimation of complexity of chaotic dynamical systems.

### Determining the Number of Samples Required to Estimate Entropy in Natural Sequences

- Computer ScienceIEEE Transactions on Information Theory
- 2019

A method is proposed for estimating the number of samples required to compute Shannon entropy for a set of ranked symbolic “natural” events using a modified Zipf-Mandelbrot law and the Dvoretzky–Kiefer–Wolfowitz inequality.

### Batch and stream entropy with fixed partitions for chaos-based random bit generators.

- Computer Science, PhysicsPhysical review. E
- 2021

Measures are proposed for reliably estimating the entropy of bits produced in an entropy source using a chaotic physical system and depend on the end-to-end method of transfer of entropy from the chaoticphysical system to the bit entropy source.

### Selecting an Effective Entropy Estimator for Short Sequences of Bits and Bytes with Maximum Entropy

- Computer ScienceEntropy
- 2021

In this work, the most effective estimator is selected to estimate entropy in short samples of bytes and bits with maximum entropy, based on its bias, the mean square error long samples ofbytes and bits.

### Large Deviations for Empirical Entropies of Gibbsian sources

- Computer Science
- 2004

The entropy of an ergodic finite-alphabet process can be computed from a single typical sample path x1 using the entropy of the k-block empirical probability and letting k grow with n roughly like…

### Regularities unseen, randomness observed: levels of entropy convergence.

- Computer ScienceChaos
- 2003

Several phenomenological approaches to applying information theoretic measures of randomness and memory to stochastic and deterministic processes are synthesized by using successive derivatives of the Shannon entropy growth curve to look at the relationships between a process's entropy convergence behavior and its underlying computational structure.

### Fluctuations of the Empirical Entropies of a Chain of Infinite Order

- Mathematics, Computer Science
- 2003

This paper addresses the question of the fluctuations of the empirical entropy of a chain of infinite order and proves a central limit theorem for two possible definitions, both based on the empirical distribution of cylinders with length clog n.

## References

SHOWING 1-10 OF 75 REFERENCES

### Estimating the information content of symbol sequences and efficient codes

- Computer ScienceIEEE Trans. Inf. Theory
- 1989

Several variants of an algorithm for estimating Shannon entropies of symbol sequences are presented, which seem to be the optimal algorithms for sequences with strong long-range correlations, e.g. natural languages.

### A New Method to Calculate Higher-Order Entropies from Finite Samples

- Mathematics
- 1993

The finite size of natural symbol sequences, e.g. DNA strands or texts in natural languages, is responsible for the worsening statistics with respect to increasing length n of the substrings, thus…

### Compression of individual sequences via variable-rate coding

- Computer ScienceIEEE Trans. Inf. Theory
- 1978

The proposed concept of compressibility is shown to play a role analogous to that of entropy in classical information theory where one deals with probabilistic ensembles of sequences rather than with individual sequences.

### Entropy and Long-Range Correlations in Literary English

- Physics
- 1994

We investigated long-range correlations in two literary texts, "Moby Dyck" by H. Melville and Grimm's tales. The analysis is based on the calculation of entropylike quantities as the mutual…

### Entropy of symbolic sequences: the role of correlations

- Computer Science
- 1991

It is shown that sporadic systems give rise to peculiar scaling properties as a result of long-range correlations, and the potential implications of this possibility in the structure of natural languages are explored.

### Enlarged scaling ranges for the KS-entropy and the information dimension.

- Computer ScienceChaos
- 1996

Rewriting differences of block entropies as averages over decay rates, and ignoring all parts of the sample where these rates are uncomputable because of the lack of neighbours, yields improved entropy estimates, and the scaling range for estimates of the information dimension can be extended considerably.

### On the Complexity of Finite Sequences

- MathematicsIEEE Trans. Inf. Theory
- 1976

A new approach to the problem of evaluating the complexity ("randomness") of finite sequences is presented, related to the number of steps in a self-delimiting production process by which a given sequence is presumed to be generated.

### A universal data compression system

- Computer ScienceIEEE Trans. Inf. Theory
- 1983

A universal data compression algorithm is described which is capable of compressing long strings generated by a "finitely generated" source, with a near optimum per symbol length without prior…