An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory

  title={An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory},
  author={Neri Merhav},
  journal={2008 IEEE International Symposium on Information Theory},
  • N. Merhav
  • Published 17 February 2007
  • Mathematics, Computer Science
  • 2008 IEEE International Symposium on Information Theory
An identity between two versions of the large deviations rate function of the probability a certain rare event is established. This identity has an interpretation in statistical physics, namely, an isothermal equilibrium of a composite system that consists of multiple subsystems. Several information-theoretic application examples, where the analysis of this large deviations probability naturally arises, are then described from the viewpoint of this statistical mechanical interpretation. This… 
Information Theory and Statistical Physics
Relationships between information theory and statistical physics have been recognized over the last few decades. One such aspect is identifying structures of optimization problems pertaining to
Another Look at the Physics of Large Deviations With Application to Rate-Distortion Theory
  • N. Merhav
  • Mathematics, Computer Science
  • 2009
This work revisits and extends the physical interpretation recently given to a certain identity between large--deviations rate--functions, as an instance of thermal equilibrium between several physical systems that are brought into contact, and shows a new interpretation of mechanical equilibrium between these systems.
Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions
We consider the problem of signal estimation (denoising) from a statistical mechanical perspective, using a relationship between the minimum mean square error (MMSE), of estimating a signal, and the
On the Statistical Physics of Directed Polymers in a Random Medium and Their Relation to Tree Codes
  • N. Merhav
  • Mathematics, Computer Science
    IEEE Transactions on Information Theory
  • 2010
It is proved that random tree codes achieve the distortion-rate function, not only on the average, but moreover, almost surely under a certain symmetry condition.
Rate–Distortion Function via Minimum Mean Square Error Estimation
  • N. Merhav
  • Mathematics, Computer Science
    IEEE Transactions on Information Theory
  • 2011
It is demonstrated that the new relations among rate, distortion, and MMSE can sometimes be rather useful for obtaining non-trivial upper and lower bounds on the rate-distortion function, as well as for determining the exact asymptotic behavior for very low and for very large distortion.
Shannon’s rate-distortion curve characterizes optimal lossy compression. I show here that the optimization principle that has to be solved to compute the rate-distortion function can be derived from


A Statistical Mechanics Approach to Large Deviation Theorems
Chernoo bounds and related large deviation bounds have a wide variety of applications in statistics and learning theory. This paper proves that for any real-valued random variable X the probability
Information Theory and Statistical Mechanics
Treatment of the predictive aspect of statistical mechanics as a form of statistical inference is extended to the density-matrix formalism and applied to a discussion of the relation between
Mapping of statistical physics to information theory with application to biological systems.
The meaning of the Handscomb Monte-Carlo method is extended to a general recipe for the transformation from a "configuration" space to a "sentence" space and a possible mapping procedure based on a generalization of the handscomb representation is described.
The Nishimori line and Bayesian statistics
  • Y. Iba
  • Mathematics, Physics
  • 1999
The `Nishimori line' is a line or hypersurface in the parameter space of systems with quenched disorder, where simple expressions of the averages of physical quantities over the quenched random
The physics of forgetting: Landauer's erasure principle and information theory
This article discusses the concept of information and its intimate relationship with physics. After an introduction of all the necessary quantum mechanical and information theoretical concepts we
A new class of upper bounds on the log partition function
A new class of upper bounds on the log partition function of a Markov random field (MRF) is introduced, based on concepts from convex duality and information geometry, and the Legendre mapping between exponential and mean parameters is exploited.
Statistical mechanics of error exponents for error-correcting codes
  • Thierry Mora, O. Rivoire
  • Mathematics, Physics
    Physical review. E, Statistical, nonlinear, and soft matter physics
  • 2006
A general, thermodynamic, formalism is introduced that illustrates maximum-likelihood decoding of low-density parity-check codes on the binary erasure channel and the binary symmetric channel and applies the cavity method for large deviations to derive expressions for both the average and typical error exponents.
Statistical mechanics of typical set decoding.
The performance of "typical set (pairs) decoding" for ensembles of Gallager's linear code is investigated using statistical physics and it is shown that the average error rate for the second type of error over a given code ensemble can be accurately evaluated using the replica method.
Spin Glasses, Error-Correcting Codes and Finite-Temperature Decoding
The probability for a sequence of information symbols to have been sent, when the transmission channel's output is known, is simply related to a spin glass Hamiltonian. The ground state of this
A mapping approach to rate-distortion computation and analysis
  • K. Rose
  • Mathematics, Computer Science
    IEEE Trans. Inf. Theory
  • 1994
This paper reformulated the rate-distortion problem in terms of the optimal mapping from the unit interval with Lebesgue measure that would induce the desired reproduction probability density and shows how the number of "symbols" grows as the system undergoes phase transitions.