# A Bayesian Characterization of Relative Entropy

@article{Baez2014ABC, title={A Bayesian Characterization of Relative Entropy}, author={John C. Baez and Tobias Fritz}, journal={ArXiv}, year={2014}, volume={abs/1402.3067} }

We give a new characterization of relative entropy, also known as the Kullback-Leibler divergence. We use a number of interesting categories related to probability theory. In particular, we consider a category FinStat where an object is a finite set equipped with a probability distribution, while a morphism is a measure-preserving function $f: X \to Y$ together with a stochastic right inverse $s: Y \to X$. The function $f$ can be thought of as a measurement process, while s provides a…

## Topics from this paper

## 46 Citations

A functorial characterization of von Neumann entropy

- Computer Science, PhysicsArXiv
- 2020

The von Neumann entropy is classified as a certain concave functor from finite-dimensional non-commutative probability spaces and state-preserving $*$-homomorphisms to real numbers and the existence of disintegrations for classical probability spaces plays a crucial role in this classification.

A Probability Monad as the Colimit of Finite Powers

- Mathematics, Computer ScienceArXiv
- 2017

It is proved that this Kantorovich monad arises from a colimit construction on finite powers, which formalizes the intuition that probability measures are limits of finite samples and allows for the development of integration theory and other things.

A Probability Monad as the Colimit of Spaces of Finite Samples

- Mathematics
- 2017

We define and study a probability monad on the category of complete metric spaces and short maps. It assigns to each space the space of Radon probability measures on it with finite first moment,…

Towards a functorial description of quantum relative entropy

- Computer Science, PhysicsGSI
- 2021

Preliminary calculations suggest that the finite-dimensional quantum (Umegaki) relative entropy might be characterized in a similar way to the classical relative entropy, and it is explicitly proved that it defines an affine functor in the special case where the relative entropy is finite.

A non-commutative Bayes' theorem.

- Mathematics, Physics
- 2020

Using a diagrammatic reformulation of Bayes' theorem, we provide a necessary and sufficient condition for the existence of Bayesian inference in the setting of finite-dimensional $C^*$-algebras. In…

Dirichlet Polynomials and Entropy

- Computer Science, MathematicsEntropy
- 2021

The main result of this paper is the following: the rectangle-area formula A(d) = L (d)W( d) holds for any Dirichlet polynomial d, and the entropy of an empirical distribution can be calculated entirely in terms of the homomorphism h applied to its corresponding Dirichlets.

Stinespring's construction as an adjunction

- Mathematics, Physics
- 2018

Given a representation of a unital $C^*$-algebra $\mathcal{A}$ on a Hilbert space $\mathcal{H}$, together with a bounded linear map $V:\mathcal{K}\to\mathcal{H}$ from some other Hilbert space, one…

The Design of Global Correlation Quantifiers and Continuous Notions of Statistical Sufficiency

- Mathematics, Computer ScienceEntropy
- 2020

Using first principles from inference, a set of functionals are found to be uniquely capable of determining whether a certain class of inferential transformations, ρ→∗ρ′, preserve, destroy or create correlations.

Non-commutative disintegrations: existence and uniqueness in finite dimensions

- Physics, Mathematics
- 2019

We utilize category theory to define non-commutative disintegrations, regular conditional probabilities, and optimal hypotheses for finite-dimensional C*-algebras. In the process, we introduce a…

Postquantum Br\`{e}gman relative entropies and nonlinear resource theories

- Mathematics
- 2017

We introduce the family of postquantum Br\`{e}gman relative entropies, based on nonlinear embeddings into reflexive Banach spaces (with examples given by reflexive noncommutative Orlicz spaces over…

## References

SHOWING 1-10 OF 24 REFERENCES

A Characterization of Entropy in Terms of Information Loss

- Mathematics, Computer ScienceEntropy
- 2011

It is shown that Shannon entropy gives the only concept of information loss that is functorial, convex-linear and continuous and naturally generalizes to Tsallis entropy as well.

Convex Spaces I: Definition and Examples

- Mathematics
- 2009

We propose an abstract definition of convex spaces as sets where one can take convex combinations in a consistent way. A priori, a convex space is an algebra over a finitary version of the Giry…

Quantum Entropy and Its Use

- Mathematics
- 1993

I Entropies for Finite Quantum Systems.- 1 Fundamental Concepts.- 2 Postulates for Entropy and Relative Entropy.- 3 Convex Trace Functions.- II Entropies for General Quantum Systems.- 4 Modular…

An Introduction to the Theory of Functional Equations and Inequalities

- Mathematics
- 2008

Preliminaries.- Set Theory.- Topology.- Measure Theory.- Algebra.- Cauchy's Functional Equation and Jensen's Inequality.- Additive Functions and Convex Functions.- Elementary Properties of Convex…

Bayesian surprise attracts human attention

- Computer Science, PsychologyVision Research
- 2009

A formal Bayesian definition of surprise is proposed to capture subjective aspects of sensory information and it is shown that Bayesian surprise is a strong attractor of human attention, with 72% of all gaze shifts directed towards locations more surprising than the average.

Higher Operads, Higher Categories

- Mathematics
- 2003

Part I. Background: 1. Classical categorical structures 2. Classical operads and multicategories 3. Notions of monoidal category Part II. Operads. 4. Generalized operads and multicategories: basics…

An operadic introduction to entropy, The n-Category Café, 18 May 2011. Available at http://golem.ph.utexas.edu/category/2011/05/an operadic introduction to en.html

- 2011

Higher Operads

- Higher Categories, London Mathematical Society Lecture Note Series 298, Cambridge U. Press, Cambridge
- 2004

An operadic introduction to entropy, The n-Category Café

- An operadic introduction to entropy, The n-Category Café
- 2011

Baldi , Bayesian surprise attracts human attention

- Advances in Neural Information Processing Systems
- 2005