Active inference and epistemic value

@article{Friston2015ActiveIA,
  title={Active inference and epistemic value},
  author={Karl J. Friston and Francesco Rigoli and Dimitri Ognibene and Christoph D Mathys and Thomas H. B. FitzGerald and Giovanni Pezzulo},
  journal={Cognitive Neuroscience},
  year={2015},
  volume={6},
  pages={187 - 214}
}
We offer a formal treatment of choice behavior based on the premise that agents minimize the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (or intrinsic) value. Minimizing expected free energy is therefore equivalent to maximizing extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximizing information gain or intrinsic value (or reducing uncertainty… 
Active inference, Bayesian optimal design, and expected utility
TLDR
This chapter describes how active inference combines Bayesian decision theory and optimal Bayesian design principles under a single imperative to minimize expected free energy to enable the natural emergence of information-seeking behavior.
Evidence for surprise minimization over value maximization in choice behavior
TLDR
It is shown that human decision-making is better explained by surprise minimization compared to utility maximization, and a limitation of purely economic motivations in explaining choice behavior is highlighted and the importance of belief-based motivations is emphasized.
Demystifying active inference
TLDR
This review disambiguates properties of active inference, by providing a condensed overview of the theory underpinning active inference and noting that this formalism can be applied in other engineering applications; e.g., robotic arm movement, playing Atari games, etc., if appropriate underlying probability distributions can be formulated.
Active Inference and Epistemic Value in Graphical Models
TLDR
It is concluded that CBFE optimization by message passing suggests a general mechanism for epistemic-aware AIF in free-form generative models, similar to how an EFE agent incurs expected reward in significantly more environmental scenarios.
Prior Preference Learning from Experts: Designing a Reward with Active Inference
Understanding the origin of information-seeking exploration in probabilistic objectives for control
TLDR
This paper proposes a dichotomy in the objective functions underlying adaptive behaviour between evidence objectives, which correspond to well-known reward or utility maximizing objectives and their novel divergence objectives which instead seek to minimize the divergence between the agent's expected and desired distribution over futures and which give rise to information-seeking exploration.
Computational mechanisms of curiosity and goal-directed exploration
TLDR
This work illustrates the emergence of different types of information-gain, termed active inference and active learning, and shows how these forms of exploration induce distinct patterns of ‘Bayes-optimal’ behaviour.
Whence the Expected Free Energy?
TLDR
This letter investigates the origins of the EFE in detail and presents a functional that is argued is the natural extension of the VFE but actively discourages exploratory behavior, thus demonstrating that exploration does not directly follow from free energy minimization into the future.
Generalised free energy and active inference: can the future cause the past?
TLDR
This work compares two free energy functionals for active inference under Markov decision processes and shows that policies are inferred or selected that realise prior preferences by minimising the free energy of future expectations.
Introducing a Bayesian model of selective attention based on active inference
TLDR
A formal model of selective attention based on active inference and contextual epistemic foraging is introduced, which shows that the atypical exploratory behaviours in conditions such as autism and anxiety may be due to a failure to appropriately modulate sensory precision in a context-specific way.
...
...

References

SHOWING 1-10 OF 190 REFERENCES
The anatomy of choice: active inference and agency
TLDR
Variational Bayes is considered as an alternative scheme that provides formal constraints on the computational anatomy of inference and action—constraints that are remarkably consistent with neuroanatomy.
The anatomy of choice: dopamine and decision-making
TLDR
Variational Bayes is considered as a scheme that the brain might use for approximate Bayesian inference that optimizes a free energy bound on model evidence and changes in precision during variational updates are remarkably reminiscent of empirical dopaminergic responses.
Active inference and agency: optimal control without cost functions
TLDR
A variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty is described, showing that optimal control can be cast as active inference and leading to a distinction between models with and without inference on hidden control states.
Exploration, novelty, surprise, and free energy minimization
TLDR
This paper reviews recent developments under the free energy principle that introduce a normative perspective on classical economic (utilitarian) decision-making based on (active) Bayesian inference and dissolves any dialectic between minimizing surprise and exploration or novelty seeking.
Observing the Observer (I): Meta-Bayesian Models of Learning and Decision-Making
TLDR
A generic approach that can be used to infer how subjects make optimal decisions under uncertainty is presented, which induces a distinction between a subject's perceptual model, which underlies the representation of a hidden “state of affairs” and a response models, which predicts the ensuing behavioural responses to those inputs.
Information, Utility and Bounded Rationality
TLDR
It is shown that bounded optimal control solutions can be derived from this variational principle, which leads in general to stochastic policies, and that risk-sensitive and robust control schemes fall out naturally if the environment is considered as a bounded rational and perfectly rational opponent, respectively.
Thermodynamics as a theory of decision-making with information-processing costs
  • Pedro A. Ortega, D. Braun
  • Economics
    Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences
  • 2013
Perfectly rational decision-makers maximize expected utility, but crucially ignore the resource costs incurred when determining optimal actions. Here, we propose a thermodynamically inspired
Information-Theoretic Bounded Rationality and ε-Optimality
TLDR
It is found that the stochastic policies that optimize the free energy trade-off comply with the notion of e-optimality, and this optimality criterion even holds when the environment is adversarial.
The Dopaminergic Midbrain Encodes the Expected Certainty about Desired Outcomes
TLDR
It is demonstrated that human subjects infer both optimal policies and the precision of those inferences, and thus support the notion that humans perform hierarchical probabilistic Bayesian inference.
Model averaging, optimal inference, and habit formation
TLDR
It is hypothesized that Bayesian model averaging plays an important role in cognition, given that it is both optimal and realizable within a plausible neuronal architecture, and a number of apparently suboptimal phenomena within the framework of approximate Bayesian inference are proposed.
...
...