Universal psychometrics: Measuring cognitive abilities in the machine kingdom
@article{HernndezOrallo2014UniversalPM, title={Universal psychometrics: Measuring cognitive abilities in the machine kingdom}, author={Jos{\'e} Hern{\'a}ndez-Orallo and David L. Dowe and Mª Victoria Hern{\'a}ndez-Lloreda}, journal={Cognitive Systems Research}, year={2014}, volume={27}, pages={50-74} }
41 Citations
On the universality of cognitive tests
- Computer ScienceArXiv
- 2013
This paper analyses the question of universal tests in terms of subjects, environments, space-time resolution, rewards and interfaces, according to several levels where universal tests may be progressively more difficult to conceive, implement and administer.
How universal can an intelligence test be?
- Computer ScienceAdapt. Behav.
- 2014
It is argued that such tests must be highly adaptive, i.e. that tasks, resolution, rewards and communication have to be adapted according to how the evaluated agent is reacting and performing, and set the quest for universal tests as a progressive rather than absolute goal.
The Measure of All Minds: Evaluating Natural and Artificial Intelligence
- Psychology
- 2017
Using algorithmic information theory as a foundation, the book elaborates on the evaluation of perceptual, developmental, social, verbal and collective features and critically analyzes what the future of intelligence might look like.
Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement
- Computer ScienceArtificial Intelligence Review
- 2016
This paper critically assess the different ways AI systems are evaluated, and the role of components and techniques in these systems, and identifies three kinds of evaluation: human discrimination, problem benchmarks and peer confrontation.
A computational analysis of general intelligence tests for evaluating cognitive development
- PsychologyCognitive Systems Research
- 2017
On the Measure of Intelligence
- Computer ScienceArXiv
- 2019
A new formal definition of intelligence is articulate, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience, and a set of guidelines for what a general AI benchmark should look like are proposed.
General intelligence disentangled via a generality metric for natural and artificial intelligence
- Computer ScienceScientific reports
- 2021
This work dissects the notion of general intelligence into two non-populational measures, generality and capability, which it applies to individuals and groups of humans, other animals and AI systems, and relates the individual measure of generality to traditional notions of general Intelligence and cognitive efficiency in humans, collectives, non-human animals and machines.
Universal Psychometrics Tasks: difficulty, composition and decomposition
- Computer Science, PsychologyArXiv
- 2015
This note revisits the concepts of task and difficulty and introduces a notion of asynchronous-time stochastic tasks, which is crucial for the determination of the difficulty of a task as the number of computational steps required to acquire an acceptable policy for the task.
Computer models solving intelligence test problems: Progress and implications
- Computer ScienceArtif. Intell.
- 2016
Is Spearman's Law of Diminishing Returns (SLODR) Meaningful for Artificial Agents?
- BiologyECAI
- 2016
This paper analyzes for the first time whether the SLODR makes sense for artificial agents and what its role should be in the analysis of general-purpose AI.
References
SHOWING 1-10 OF 115 REFERENCES
On Potential Cognitive Abilities in the Machine Kingdom
- Computer ScienceMinds and Machines
- 2012
This paper investigates the notion of potential cognitive ability for machines, focussing especially on universality and intelligence, and considers several machine characterisations (non-interactive and interactive) and gives definitions for each case, considering permanent and temporal potentials.
On the Computational Measurement of Intelligence Factors
- Computer Science
- 2011
A scale for measuring the complexity of an instance of a problem, depending on the descriptional complexity (Levin LT variant) of the ‘explanation’ of the answer to the problem is developed.
Measuring universal intelligence: Towards an anytime intelligence test
- Computer ScienceArtif. Intell.
- 2010
Beyond the Turing Test
- Computer ScienceJ. Log. Lang. Inf.
- 2000
The idea of using the Turing Test as apractical test of intelligence should be surpassed, and substituted by computational and factorial tests of different cognitive abilities, a more useful approach for artificial intelligence progress and for many other intriguing questions that present themselves beyond theTuring Test.
On More Realistic Environment Distributions for Defining, Evaluating and Developing Intelligence
- Computer ScienceAGI
- 2011
This paper agrees with previous criticisms that a universal distribution using a reference universal Turing machine (UTM) over tasks, environments, etc., is perhaps amuch too general and proposes the notion of Darwin-Wallace distribution for environments, which is inspired by biological evolution, artificial life and evolutionary computation.
Human Cognitive Abilities-a sur-vey of factor-analytic studies
- Psychology
- 1993
Preface Part I. Introduction to the Survey: 1. The study of cognitive abilities 2. Historical foundations of the study of cognitive abilities 3. Survey and analysis of correlational and…
Psychometric artificial intelligence
- Computer ScienceJ. Exp. Theor. Artif. Intell.
- 2011
The paper in the present issue of JETAI can be plausibly viewed as resurrecting this approach to machine intelligence, in the form of what is called Psychometric AI, or just PAI (rhymes with ‘ ’).
Universal Intelligence: A Definition of Machine Intelligence
- Computer ScienceMinds and Machines
- 2007
A number of well known informal definitions of human intelligence are taken, and mathematically formalised to produce a general measure of intelligence for arbitrary machines that formally captures the concept of machine intelligence in the broadest reasonable sense.
An Approximation of the Universal Intelligence Measure
- Computer ScienceAlgorithmic Probability and Friends
- 2011
This paper studies the practical issues involved in developing a real-world UIM-based performance metric and develops a prototype implementation which is used to evaluate a number of different artificial agents.