The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents

@article{Bostrom2012TheSW,
  title={The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents},
  author={Nick Bostrom},
  journal={Minds and Machines},
  year={2012},
  volume={22},
  pages={71-85}
}
  • N. Bostrom
  • Published 1 May 2012
  • Psychology, Computer Science
  • Minds and Machines
This paper discusses the relation between intelligence and motivation in artificial agents, developing and briefly arguing for two theses. The first, the orthogonality thesis, holds (with some caveats) that intelligence and final goals (purposes) are orthogonal axes along which possible artificial intellects can freely vary—more or less any level of intelligence could be combined with more or less any final goal. The second, the instrumental convergence thesis, holds that as long as they… 
Formalizing Convergent Instrumental Goals
TLDR
This paper provides formal models that demonstrate Omohundro’s thesis that sufficiently advanced AI systems of any design would, by default, have incentives to pursue a number of instrumentally useful subgoals, such as acquiring more computing power and amassing many resources.
Bounding the impact of AGI
TLDR
This work uses the method of bounds to argue that AGIs meeting these criteria are subject to Gewirth's dialectical argument to the necessity of morality, compelling them to behave in a moral fashion, provided GEWirth's argument can be formally shown to be conclusive.
Ethical Artificial Intelligence
TLDR
How future AI will differ from current AI, the politics of AI, and the ultimate use of AI to help understand the nature of the universe and the authors' place in it are discussed.
Dynamic Cognition Applied to Value Learning in Artificial Intelligence
TLDR
It is of utmost importance that artificial intelligent agents have their values aligned with human values, given the fact that an AI cannot expect an AI to develop the authors' moral preferences simply because of its intelligence.
Approaches to Deploying a Safe Artificial Moral Agent
The prospect of creating beings with intelligence far beyond our own is hugely exiting. It is also hugely worrying because we don’t yet know if the immense power that we believe these agents would
Artificial Intelligence, Rationality, and the World Wide Web
  • J. Murphy
  • Computer Science
    IEEE Intelligent Systems
  • 2018
TLDR
It is argued that a third possibility is plausible yet generally overlooked: for several different reasons, an articial superintelligence might choose to exert no appreciable effect on the status quo ante (the already existing collective superintelligence of commercial cyberspace).
Dynamic Models Applied to Value Learning in Artificial Intelligence
TLDR
It is of utmost importance that artificial intelligent agents have their values aligned with human values, given the fact that the authors cannot expect an AI to develop human moral values simply because of its intelligence, as discussed in the Orthogonality Thesis.
Don't Fear the Reaper: Refuting Bostrom's Superintelligence Argument
TLDR
Concern about the possibility that an autonomous agent might modify itself to become "superintelligent" and destroy all of humanity is misplaced and better directed at policy questions around data access and storage.
Motivated Value Selection for Artificial Agents
  • S. Armstrong
  • Computer Science
    AAAI Workshop: AI and Ethics
  • 2015
TLDR
The conditions under which motivated value selection is an issue for some types of agents are established, and an example of an `indifferent' agent that avoids it entirely is presented, which poses and solves an issue which has not been formally addressed in the literature.
The Singularity and Machine Ethics
Many researchers have argued that a self-improving artificial intelligence (AI) could become so vastly more powerful than humans that we would not be able to stop it from achieving its goals. If so,
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 43 REFERENCES
Rational Artificial Intelligence for the Greater Good
Today’s technology is mostly preprogrammed but the next generation will make many decisions autonomously. This shift is likely to impact every aspect of our lives and will create many new benefits
Learning What to Value
I. J. Good's intelligence explosion theory predicts that ultraintelligent agents will undergo a process of repeated self-improvement; in the wake of such an event, how well our values are fulfilled
Complex Value Systems are Required to Realize Valuable Futures
AcommonreactiontofirstencounteringtheproblemstatementofFriendlyAI(“Ensure that the creation of a generally intelligent, self-improving, eventually superintelligent system realizes a positive
The Basic AI Drives
TLDR
This paper identifies a number of “drives” that will appear in sufficiently advanced AI systems of any design and discusses how to incorporate these insights in designing intelligent technology which will lead to a positive future for humanity.
Ontological Crises in Artificial Agents' Value Systems
TLDR
This paper discusses in this paper which sorts of agents will undergo ontological crises and why they may want to create such agents, and argues that a well-defined procedure for resolving ontology crises is needed.
Artificial Intelligence as a Positive and Negative Factor in Global Risk
By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Of course this problem is not limited to the field of AI. Jacques Monod wrote: "A
Artificial Intelligence: A Modern Approach
The long-anticipated revision of this #1 selling book offers the most comprehensive, state of the art introduction to the theory and practice of artificial intelligence for modern applications.
The Singularity: a Philosophical Analysis
What happens when machines become more intelligent than humans? One view is that this event will be followed by an explosion to ever-greater levels of intelligence, as each generation of machines
Are We Living in a Computer Simulation
This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a "posthuman" stage; (2) any posthuman civilization is
What matters?
TLDR
It is demonstrated in this study that the T&FR Committee’s activities to examine taxation matters conform to the three key institutional traits (dependence upon the state, monopolistic representative position and bureaucratically hierarchical structure) provided by Williamson's general model.
...
1
2
3
4
5
...