• Corpus ID: 16483331

Toward Idealized Decision Theory

  title={Toward Idealized Decision Theory},
  author={Nate Soares and Benja Fallenstein},
This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward… 

Figures and Tables from this paper

Sequential Extensions of Causal and Evidential Decision Theory
This work extends causal decision algorithms to the sequential setting where the agent alternates between taking actions and observing their consequences, and finds that evidential decision theory has two natural extensions while causal decision theory only has one.
Functional Decision Theory: A New Theory of Instrumental Rationality
This paper defines FDT, explores its prescriptions in a number of different decision problems, compares it to CDT and EDT, and gives philosophical justifications for FDT as a normative theory of decision-making.
Approval-directed agency and the decision theory of Newcomb-like problems
This paper studies what decision theory an approval-directed agent, i.e., an agent whose goal it is to maximize the score it receives from an overseer, implements and shows which of these two decision theories describes the agent’s behavior in which situations.
Extracting Money from Causal Decision Theorists
This paper provides a new argument against what is probably the most popular variant of expected utility maximization: causal decision theory (CDT), and provides two scenarios in which CDT voluntarily loses money.
Success-First Decision Theories
The standard formulation of Newcomb's problem compares evidential and causal conceptions of expected utility, with those maximizing evidential expected utility tending to end up far richer. Thus, in
Epistemic Time Bias in Newcomb’s Problem∗
Causal decision theorists like David Lewis hold that, while an agent should choose acts by using her current, rather than anticipated, credences over causal dependency hypotheses, she should also
Cheating Death in Damascus
Evidential Decision Theory (EDT) and Causal Decision Theory (CDT) are the leading contenders as theories of rational action, but both face counterexamples. We present some new counterexamples,
Plausibility and probability in deductive reasoning
A normative model of fair bets under deductive uncertainty which draws from both probability and the theory of algorithms is built, using Bayesian-inspired arguments.
A typology of Newcomblike problems
This paper introduces a typology of Newcomblike problems—decision problems in which evidential decision theory and causal decision theory come apart. Such problems involve an evidential dependence
Quantilizers: A Safer Alternative to Maximizers for Limited Optimization
This paper describes an alternative to expected utility maximization for powerful AI systems, which is called expected utility quantilization, which could allow the construction of AI systems that do not necessarily fall into strange and unanticipated shortcuts and edge cases in pursuit of their goals.


Reversing 30 years of discussion: why causal decision theorists should one-box
The paper will show how one may rationalize one-boxing in Newcomb’s problem and drinking the toxin in the Toxin puzzle within the confines of causal decision theory by ascending to so-called
The Foundations of Causal Decision Theory
A chance to reconsider Prudential rationality as expected utility maximization as well as a representation theorem for causal decision theory.
Causal decision theory
My own version of causal decision theory is given, compared with versions offered by several other authors, and it is suggested that the versions have more in common than meets the eye.
Evidential Decision Theory and Medical Newcomb Problems
  • Arif Ahmed
  • Philosophy
    The British Journal for the Philosophy of Science
  • 2005
Huw Price ([1991]) has offered evidential decision theorists a defence against the charge that they make unintuitive recommendations for cases like Newcomb's Problem. He says that when conditional
Metatickles and the dynamics of deliberation
The traditional or “orthodox” decision rule of maximizing conditional expected utility has recently come under attack by critics who advance alternative “causal” decision theories. The traditional
Binding and its consequences
In “Bayesianism, Infinite Decisions, and Binding”, Arntzenius et al. (Mind 113:251–283, 2004) present cases in which agents who cannot bind themselves are driven by standard decision theory to choose
Gandalf’s solution to the Newcomb problem
It is argued that BT gives the right verdict on the cases that seem to be counterexamples to CDT and EDT, and gives a prominent role to the notion of a “benchmark” for each state of nature, by comparison with which the value of the available options within states of nature are measured.
Causal Decision Theory
  • E. Eells
  • Philosophy
    PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association
  • 1984
After a brief presentation of evidential decision theory, causal decision theory, and Newcomb type prima facie counterexamples to the evidential theory, three kinds of "metatickle" defenses of the
Counterfactuals and Two Kinds of Expected Utility
We begin with a rough theory of rational decision-making. In the first place, rational decision-making involves conditional propositions: when a person weighs a major decision, it is rational for him
Some Counterexamples to Causal Decision Theory
Many philosophers have been converted to causal decision theory by something like the following line of argument: Evidential decision theory endorses irrational courses of action in a range of