Corpus ID: 669957

A Bayesian Characterization of Relative Entropy

  title={A Bayesian Characterization of Relative Entropy},
  author={John C. Baez and Tobias Fritz},
  • J. Baez, T. Fritz
  • Published 13 February 2014
  • Mathematics, Computer Science, Physics
  • ArXiv
We give a new characterization of relative entropy, also known as the Kullback-Leibler divergence. We use a number of interesting categories related to probability theory. In particular, we consider a category FinStat where an object is a finite set equipped with a probability distribution, while a morphism is a measure-preserving function $f: X \to Y$ together with a stochastic right inverse $s: Y \to X$. The function $f$ can be thought of as a measurement process, while s provides a… Expand
A functorial characterization of von Neumann entropy
The von Neumann entropy is classified as a certain concave functor from finite-dimensional non-commutative probability spaces and state-preserving $*$-homomorphisms to real numbers and the existence of disintegrations for classical probability spaces plays a crucial role in this classification. Expand
A Probability Monad as the Colimit of Finite Powers
It is proved that this Kantorovich monad arises from a colimit construction on finite powers, which formalizes the intuition that probability measures are limits of finite samples and allows for the development of integration theory and other things. Expand
A Probability Monad as the Colimit of Spaces of Finite Samples
We define and study a probability monad on the category of complete metric spaces and short maps. It assigns to each space the space of Radon probability measures on it with finite first moment,Expand
Towards a functorial description of quantum relative entropy
Preliminary calculations suggest that the finite-dimensional quantum (Umegaki) relative entropy might be characterized in a similar way to the classical relative entropy, and it is explicitly proved that it defines an affine functor in the special case where the relative entropy is finite. Expand
A non-commutative Bayes' theorem.
Using a diagrammatic reformulation of Bayes' theorem, we provide a necessary and sufficient condition for the existence of Bayesian inference in the setting of finite-dimensional $C^*$-algebras. InExpand
Stinespring's construction as an adjunction
Given a representation of a unital $C^*$-algebra $\mathcal{A}$ on a Hilbert space $\mathcal{H}$, together with a bounded linear map $V:\mathcal{K}\to\mathcal{H}$ from some other Hilbert space, oneExpand
The Design of Global Correlation Quantifiers and Continuous Notions of Statistical Sufficiency
Using first principles from inference, a set of functionals are found to be uniquely capable of determining whether a certain class of inferential transformations, ρ→∗ρ′, preserve, destroy or create correlations. Expand
Non-commutative disintegrations: existence and uniqueness in finite dimensions
We utilize category theory to define non-commutative disintegrations, regular conditional probabilities, and optimal hypotheses for finite-dimensional C*-algebras. In the process, we introduce aExpand
Postquantum Br\`{e}gman relative entropies and nonlinear resource theories
We introduce the family of postquantum Br\`{e}gman relative entropies, based on nonlinear embeddings into reflexive Banach spaces (with examples given by reflexive noncommutative Orlicz spaces overExpand
An axiomatic characterization of mutual information
We characterize mutual information as the unique map on ordered pairs of random variables satisfying a set of axioms similar to those of Faddeev’s characterization of the Shannon entropy. There is aExpand


A Characterization of Entropy in Terms of Information Loss
It is shown that Shannon entropy gives the only concept of information loss that is functorial, convex-linear and continuous and naturally generalizes to Tsallis entropy as well. Expand
Convex Spaces I: Definition and Examples
We propose an abstract definition of convex spaces as sets where one can take convex combinations in a consistent way. A priori, a convex space is an algebra over a finitary version of the GiryExpand
Quantum Entropy and Its Use
I Entropies for Finite Quantum Systems.- 1 Fundamental Concepts.- 2 Postulates for Entropy and Relative Entropy.- 3 Convex Trace Functions.- II Entropies for General Quantum Systems.- 4 ModularExpand
An Introduction to the Theory of Functional Equations and Inequalities
Preliminaries.- Set Theory.- Topology.- Measure Theory.- Algebra.- Cauchy's Functional Equation and Jensen's Inequality.- Additive Functions and Convex Functions.- Elementary Properties of ConvexExpand
Bayesian surprise attracts human attention
A formal Bayesian definition of surprise is proposed to capture subjective aspects of sensory information and it is shown that Bayesian surprise is a strong attractor of human attention, with 72% of all gaze shifts directed towards locations more surprising than the average. Expand
Higher Operads, Higher Categories
Part I. Background: 1. Classical categorical structures 2. Classical operads and multicategories 3. Notions of monoidal category Part II. Operads. 4. Generalized operads and multicategories: basicsExpand
An operadic introduction to entropy, The n-Category Café, 18 May 2011. Available at operadic introduction to en.html
  • 2011
Higher Operads
  • Higher Categories, London Mathematical Society Lecture Note Series 298, Cambridge U. Press, Cambridge
  • 2004
An operadic introduction to entropy, The n-Category Café
  • An operadic introduction to entropy, The n-Category Café
  • 2011
Baldi , Bayesian surprise attracts human attention
  • Advances in Neural Information Processing Systems
  • 2005