A new class of upper bounds on the log partition function

@article{Wainwright2005ANC,
  title={A new class of upper bounds on the log partition function},
  author={M. Wainwright and T. Jaakkola and A. Willsky},
  journal={IEEE Transactions on Information Theory},
  year={2005},
  volume={51},
  pages={2313-2335}
}
We introduce a new class of upper bounds on the log partition function of a Markov random field (MRF). This quantity plays an important role in various contexts, including approximating marginal distributions, parameter estimation, combinatorial enumeration, statistical decision theory, and large-deviations bounds. Our derivation is based on concepts from convex duality and information geometry: in particular, it exploits mixtures of distributions in the exponential domain, and the Legendre… Expand
Semidefinite methods for approximate inference on graphs with cycles
We present a new method for calculating approximate marginals for probability distributions defined by graphs with cycles, based on a Gaussian entropy bound combined with a semidefinite outer boundExpand
Semidefinite Relaxations for Approximate Inference on Graphs with Cycles
We present a new method for calculating approximate marginals for probability distributions defined by graphs with cycles, based on a Gaussian entropy bound combined with a semidefinite outer boundExpand
A convex upper bound on the log-partition function for binary graphical models
We consider the problem of bounding from above the log-partition function corresponding to second-order Ising models for binary distributions. We introduce a new bound, the cardinality bound, whichExpand
New closed-form bounds on the partition function
TLDR
It is proved that TRW-BP converges in a single iteration for associative potentials, and give a closed form for the solution it finds, and a novel lower bound, in spite of being closed-form and much faster to compute, outperforms more complicated popular algorithms for computing lower bounds. Expand
How to calculate partition functions using convex programming hierarchies: provable bounds for variational methods
TLDR
This work considers dense and low threshold rank graphs, and interestingly, the reason the approach works on these types of graphs is because local correlations propagate to global correlations – completely the opposite of algorithms based on correlation decay. Expand
MAP estimation via agreement on (hyper)trees: Message-passing and linear programming
TLDR
This work develops and analyzes methods for computing provably optimal MAP configurations for a subclass of Markov random fields defined on graphs with cycles, and establishes a connection between a certain LP relaxation of the mode-finding problem, and a reweighted form of the max-products message-passing algorithm. Expand
Log-determinant relaxation for approximate inference in discrete Markov random fields
TLDR
A novel method, applicable to discrete-valued Markov random fields on arbitrary graphs, for approximately solving this marginalization problem, and finds that the performance of this log-determinant relaxation is comparable or superior to the widely used sum-product algorithm over a range of experimental conditions. Expand
Entropy bounds for a Markov random subfield
  • M. Reyes, D. Neuhoff
  • Mathematics, Computer Science
  • 2009 IEEE International Symposium on Information Theory
  • 2009
TLDR
This work uses the Maximum Entropy principle and a well-known result from information geometry to show that the marginal entropy of XU is upper bounded by the MRF on the induced subgraph with moments matching the marginal distribution. Expand
MAP estimation via agreement on trees: message-passing and linear programming
TLDR
This work develops and analyze methods for computing provably optimal maximum a posteriori probability (MAP) configurations for a subclass of Markov random fields defined on graphs with cycles and establishes a connection between a certain LP relaxation of the mode-finding problem and a reweighted form of the max-product (min-sum) message-passing algorithm. Expand
New Closed-Form Bounds on the Partition Function
TLDR
It is proved that both LBP and TRW-BP converge in a single iteration and closed-form expressions for the TRW and LBP approximations to the partition function are obtained, and novel lover bounds beat popular existing bounds like the mean-field bound on densely connected graphs. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 59 REFERENCES
Stochastic approximation algorithms for partition function estimation of Gibbs random fields
We present an analysis of previously proposed Monte Carlo algorithms for estimating the partition function of a Gibbs random field. We show that this problem reduces to estimating one or moreExpand
Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching
TLDR
A class of local message-passing algorithms is developed, which is called tree-reweighted belief propagation, for efficiently computing the value of these upper bounds on the log partition function of an arbitrary undirected graphical model, as well as the associated pseudomarginals. Expand
Approximations with Reweighted Generalized Belief Propagation
TLDR
This work increases the complexity of the approximating distributions by taking combinations of junction trees, leading to a convexified Kikuchi free energy, which is minimized by reweighted generalized belief propagation. Expand
Stochastic processes on graphs with cycles: geometric and variational approaches
TLDR
A method for performing exact estimation of Gaussian processes on graphs with cycles by solving a sequence of modi ed problems on embedded spanning trees, and a tree-based reparameterization framework for approximate estimation of discrete processes are presented. Expand
Information geometry on hierarchy of probability distributions
  • S. Amari
  • Mathematics, Computer Science
  • IEEE Trans. Inf. Theory
  • 2001
TLDR
The orthogonal decomposition of an exponential family or mixture family of probability distributions has a natural hierarchical structure is given and is important for extracting intrinsic interactions in firing patterns of an ensemble of neurons and for estimating its functional connections. Expand
Maximum likelihood Markov networks : an algorithmic approach
We show how a graphical model learning problem can be presented as a purely combinatorial problem. This allows us to analyze the computational hardness of the learning problem, and devise globalExpand
Approximating discrete probability distributions with dependence trees
TLDR
It is shown that the procedure derived in this paper yields an approximation of a minimum difference in information when applied to empirical observations from an unknown distribution of tree dependence, and the procedure is the maximum-likelihood estimate of the distribution. Expand
Tree-based reparameterization framework for analysis of sum-product and related algorithms
We present a tree-based reparameterization (TRP) framework that provides a new conceptual view of a large class of algorithms for computing approximate marginals in graphs with cycles. This classExpand
Differential Geometry of Curved Exponential Families-Curvatures and Information Loss
The differential-geometrical framework is given for analyzing statistical problems related to multi-parameter families of distributions. The dualistic structures of the exponential families andExpand
Approximate Inference and Constrained Optimization
TLDR
A class of algorithms is described that solves this typically non-convex constrained minimization problem through a sequence of conveX constrained minimizations of upper bounds on the Kikuchi free energy. Expand
...
1
2
3
4
5
...