• Corpus ID: 229923541

Likelihood Ratio Exponential Families

@article{Brekelmans2020LikelihoodRE,
  title={Likelihood Ratio Exponential Families},
  author={Rob Brekelmans and Frank Nielsen and Alireza Makhzani and A. G. Galstyan and Greg Ver Steeg},
  journal={ArXiv},
  year={2020},
  volume={abs/2012.15480}
}

• Rob Brekelmans, F. Nielsen, +2 authors G. V. Steeg
• Published 31 December 2020
• Computer Science
• ArXiv

The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints [1], while the geometric mixture path is common in MCMC methods such as annealed importance sampling (AIS) [2, 3]. Linking these two ideas, recent work [4] has interpreted the geometric mixture path as an exponential family of distributions to analyse the thermodynamic variational objective (TVO) [5]. In this work, we extend likelihood ratio exponential families to include solutions to rate-distortion (RD) optimization [6, 7], the Information Bottleneck (IB) method [8], and recent ‘RDC’ approaches which combine RD and IB [9, 10]. This provides a common mathematical framework for understanding these methods via the conjugate duality of exponential families and hypothesis testing. Further, we collect existing results [11–14] to provide a variational representation of intermediate RD or TVO distributions as a minimizing an expectation of KL divergences. This solution also corresponds to a size-power tradeoff using the likelihood ratio test and the Neyman Pearson lemma. In thermodynamic integration (TI) bounds [15, 16] such as the TVO, we identify the intermediate distribution whose expected sufficient statistics match the log partition function.