# An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory

@article{Merhav2008AnIO, title={An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory}, author={Neri Merhav}, journal={2008 IEEE International Symposium on Information Theory}, year={2008}, pages={499-503} }

An identity between two versions of the large deviations rate function of the probability a certain rare event is established. This identity has an interpretation in statistical physics, namely, an isothermal equilibrium of a composite system that consists of multiple subsystems. Several information-theoretic application examples, where the analysis of this large deviations probability naturally arises, are then described from the viewpoint of this statistical mechanical interpretation. This…

## Figures and Topics from this paper

## 6 Citations

Information Theory and Statistical Physics

- 2009

Relationships between information theory and statistical physics have been recognized over the last few decades. One such aspect is identifying structures of optimization problems pertaining to…

Another Look at the Physics of Large Deviations With Application to Rate-Distortion Theory

- Mathematics, Computer ScienceArXiv
- 2009

This work revisits and extends the physical interpretation recently given to a certain identity between large--deviations rate--functions, as an instance of thermal equilibrium between several physical systems that are brought into contact, and shows a new interpretation of mechanical equilibrium between these systems.

Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions

- Mathematics, Computer ScienceIEEE Transactions on Information Theory
- 2010

We consider the problem of signal estimation (denoising) from a statistical mechanical perspective, using a relationship between the minimum mean square error (MMSE), of estimating a signal, and the…

On the Statistical Physics of Directed Polymers in a Random Medium and Their Relation to Tree Codes

- Mathematics, Computer ScienceIEEE Transactions on Information Theory
- 2010

It is proved that random tree codes achieve the distortion-rate function, not only on the average, but moreover, almost surely under a certain symmetry condition.

Rate–Distortion Function via Minimum Mean Square Error Estimation

- Mathematics, Computer ScienceIEEE Transactions on Information Theory
- 2011

It is demonstrated that the new relations among rate, distortion, and MMSE can sometimes be rather useful for obtaining non-trivial upper and lower bounds on the rate-distortion function, as well as for determining the exact asymptotic behavior for very low and for very large distortion.

LOSSY IS LAZY

- 2015

Shannon’s rate-distortion curve characterizes optimal lossy compression. I show here that the optimization principle that has to be solved to compute the rate-distortion function can be derived from…

## References

SHOWING 1-10 OF 68 REFERENCES

A Statistical Mechanics Approach to Large Deviation Theorems

- 2007

Chernoo bounds and related large deviation bounds have a wide variety of applications in statistics and learning theory. This paper proves that for any real-valued random variable X the probability…

Information Theory and Statistical Mechanics

- Mathematics
- 1957

Treatment of the predictive aspect of statistical mechanics as a form of statistical inference is extended to the density-matrix formalism and applied to a discussion of the relation between…

Mapping of statistical physics to information theory with application to biological systems.

- Mathematics, MedicineJournal of theoretical biology
- 2000

The meaning of the Handscomb Monte-Carlo method is extended to a general recipe for the transformation from a "configuration" space to a "sentence" space and a possible mapping procedure based on a generalization of the handscomb representation is described.

The Nishimori line and Bayesian statistics

- Mathematics, Physics
- 1999

The `Nishimori line' is a line or hypersurface in the parameter space of systems with quenched disorder, where simple expressions of the averages of physical quantities over the quenched random…

The physics of forgetting: Landauer's erasure principle and information theory

- Physics
- 2001

This article discusses the concept of information and its intimate relationship with physics. After an introduction of all the necessary quantum mechanical and information theoretical concepts we…

A new class of upper bounds on the log partition function

- Computer Science, MathematicsIEEE Transactions on Information Theory
- 2005

A new class of upper bounds on the log partition function of a Markov random field (MRF) is introduced, based on concepts from convex duality and information geometry, and the Legendre mapping between exponential and mean parameters is exploited.

Statistical mechanics of error exponents for error-correcting codes

- Mathematics, PhysicsPhysical review. E, Statistical, nonlinear, and soft matter physics
- 2006

A general, thermodynamic, formalism is introduced that illustrates maximum-likelihood decoding of low-density parity-check codes on the binary erasure channel and the binary symmetric channel and applies the cavity method for large deviations to derive expressions for both the average and typical error exponents.

Statistical mechanics of typical set decoding.

- Mathematics, MedicinePhysical review. E, Statistical, nonlinear, and soft matter physics
- 2002

The performance of "typical set (pairs) decoding" for ensembles of Gallager's linear code is investigated using statistical physics and it is shown that the average error rate for the second type of error over a given code ensemble can be accurately evaluated using the replica method.

Spin Glasses, Error-Correcting Codes and Finite-Temperature Decoding

- Physics
- 1994

The probability for a sequence of information symbols to have been sent, when the transmission channel's output is known, is simply related to a spin glass Hamiltonian. The ground state of this…

A mapping approach to rate-distortion computation and analysis

- Mathematics, Computer ScienceIEEE Trans. Inf. Theory
- 1994

This paper reformulated the rate-distortion problem in terms of the optimal mapping from the unit interval with Lebesgue measure that would induce the desired reproduction probability density and shows how the number of "symbols" grows as the system undergoes phase transitions.