Responses to catastrophic AGI risk: a survey

@article{Sotala2014ResponsesTC,
  title={Responses to catastrophic AGI risk: a survey},
  author={Kaj Sotala and Roman V Yampolskiy},
  journal={Physica Scripta},
  year={2014},
  volume={90}
}
Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale (‘catastrophic risk’). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that… 

Responses to Catastrophic AGI Risk : A Survey Kaj Sotala Machine Intelligence Research Institute

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk

Artificial Intelligence : Global Risk and Long Term Strategy for Ending Fast Take off

The effort in this paper tries to make cautious headway in finding the problem, evaluating predictions on the future of AI, proposing ways to ensure that Artificial Intelligence will be useful to humans – and logically evaluating such proposals.

Responses to the Journey to the Singularity

This chapter surveys various responses that have been made to the possibility of Artificial General Intelligence (AGI) possibly posing a catastrophic risk to humanity and concludes by reviewing the proposals which it feels are worthy of further study.

Artificial General Intelligence

The AGI containment problem is surveyed – the question of how to build a container in which tests can be conducted safely and reliably, even on AGIs with unknown motivations and capabilities that could be dangerous.

Risks of the Journey to the Singularity

This paper argues that humanity will create artificial general intelligence within the next twenty to one hundred years, and that individual AGIs would be capable of learning to operate in a wide variety of domains, including ones they had not been specifically designed for.

Editorial: Risks of Artificial Intelligence

This volume is the first edited volume on the risks of AI and it originates from the first confer- ence on risks ofAI (AGI-Impacts), where the current and past presidents of the AAAI wrote a short statement that "AI doomsday scenarios belong more in the realm of science fiction than science fact".

A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy

Artificial general intelligence (AGI) is AI that can reason across a wide range of domains. It has long been considered the “grand dream” or “holy grail” of AI. It also poses major issues of ethics,

AGI Safety Literature Review

The intention of this paper is to provide an easily accessible and up-to-date collection of references for the emerging field of AGI safety, and to review the current public policy on AGI.

Classification of global catastrophic risks connected with artificial intelligence

It is shown that at each level of AI’s intelligence power, separate types of possible catastrophes dominate, and that AI safety theory is complex and must be customized for each AI development level.

Disjunctive Scenarios of Catastrophic AI Risk

  • Kaj Sotala
  • Computer Science
    Artificial Intelligence Safety and Security
  • 2018
Working in security requires what has been termed “the security mindset”: an ability to look at an existing system and see how it might be compromised by a determined attacker, rather than just assuming that a plausible-sounding idea for why things could go right is sufficient to make things safe.
...

References

SHOWING 1-10 OF 399 REFERENCES

Decision Support for Safe AI Design

This paper shows that the most probable finite stochastic program to explain a finite history is finitely computable, and that there is an agent that makes such a computation without any unintended instrumental actions.

Intelligence Explosion: Evidence and Import

This chapter reviews the evidence for and against claims that there is a substantial chance the authors will create human-level AI before 2100, and recommendations for increasing the odds of a controlled intelligence explosion relative to an uncontrolled intelligence explosion.

Bounding the impact of AGI

This work uses the method of bounds to argue that AGIs meeting these criteria are subject to Gewirth's dialectical argument to the necessity of morality, compelling them to behave in a moral fashion, provided GEWirth's argument can be formally shown to be conclusive.

Avoiding Unintended AI Behaviors

This paper shows that the most probable finite stochastic program to model a finite history is finitely computable, and that there is an agent that makes such a computation without any unintended instrumental actions.

Safety Engineering for Artificial General Intelligence

It is argued that attempts to attribute moral agency and assign rights to all intelligent machines are misguided, whether applied to infrahuman or superhuman AIs, and a new science of safety engineering for intelligent artificial agents based on maximizing for what humans value is proposed.

Global Catastrophic Risks

Acknowledgements Foreword Introduction I BACKGROUND Long-term astrophysical processes Evolution theory and the future of humanity Millenial tendencies in responses to apocalyptic threats Cognitive

The Basic AI Drives

This paper identifies a number of “drives” that will appear in sufficiently advanced AI systems of any design and discusses how to incorporate these insights in designing intelligent technology which will lead to a positive future for humanity.

Artificial Intelligence as a Positive and Negative Factor in Global Risk

By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Of course this problem is not limited to the field of AI. Jacques Monod wrote: "A

Revisiting Asimov's First Law: A Response to the Call to Arms

This paper introduces four types of safety constraints that forbid or require certain agent behaviors, and presents a novel algorithm that enforces obedience of such constraints by modifying standard MDP algorithms for generating optimal policies.

Thinking Inside the Box: Controlling and Using an Oracle AI

This paper analyzes and critique various methods of controlling the AI, and suggests that an Oracle AI might be safer than unrestricted AI, but still remains potentially dangerous.
...