Universal psychometrics: Measuring cognitive abilities in the machine kingdom

@article{HernndezOrallo2014UniversalPM,
  title={Universal psychometrics: Measuring cognitive abilities in the machine kingdom},
  author={Jos{\'e} Hern{\'a}ndez-Orallo and David L. Dowe and Mª Victoria Hern{\'a}ndez-Lloreda},
  journal={Cognitive Systems Research},
  year={2014},
  volume={27},
  pages={50-74}
}
View on Elsevier
riunet.upv.es
41 Citations
Highly Influential Citations
2
Background Citations
19
Methods Citations
8
Results Citations
1

Figures and Tables from this paper

41 Citations

Citation Type

Citation Type

On the universality of cognitive tests
TLDR
This paper analyses the question of universal tests in terms of subjects, environments, space-time resolution, rewards and interfaces, according to several levels where universal tests may be progressively more difficult to conceive, implement and administer.
How universal can an intelligence test be?
TLDR
It is argued that such tests must be highly adaptive, i.e. that tasks, resolution, rewards and communication have to be adapted according to how the evaluated agent is reacting and performing, and set the quest for universal tests as a progressive rather than absolute goal.
The Measure of All Minds: Evaluating Natural and Artificial Intelligence
TLDR
Using algorithmic information theory as a foundation, the book elaborates on the evaluation of perceptual, developmental, social, verbal and collective features and critically analyzes what the future of intelligence might look like.
Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement
TLDR
This paper critically assess the different ways AI systems are evaluated, and the role of components and techniques in these systems, and identifies three kinds of evaluation: human discrimination, problem benchmarks and peer confrontation.
A computational analysis of general intelligence tests for evaluating cognitive development
On the Measure of Intelligence
TLDR
A new formal definition of intelligence is articulate, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience, and a set of guidelines for what a general AI benchmark should look like are proposed.
General intelligence disentangled via a generality metric for natural and artificial intelligence
TLDR
This work dissects the notion of general intelligence into two non-populational measures, generality and capability, which it applies to individuals and groups of humans, other animals and AI systems, and relates the individual measure of generality to traditional notions of general Intelligence and cognitive efficiency in humans, collectives, non-human animals and machines.
Universal Psychometrics Tasks: difficulty, composition and decomposition
TLDR
This note revisits the concepts of task and difficulty and introduces a notion of asynchronous-time stochastic tasks, which is crucial for the determination of the difficulty of a task as the number of computational steps required to acquire an acceptable policy for the task.
Computer models solving intelligence test problems: Progress and implications
Is Spearman's Law of Diminishing Returns (SLODR) Meaningful for Artificial Agents?
TLDR
This paper analyzes for the first time whether the SLODR makes sense for artificial agents and what its role should be in the analysis of general-purpose AI.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 115 REFERENCES
On Potential Cognitive Abilities in the Machine Kingdom
TLDR
This paper investigates the notion of potential cognitive ability for machines, focussing especially on universality and intelligence, and considers several machine characterisations (non-interactive and interactive) and gives definitions for each case, considering permanent and temporal potentials.
On the Computational Measurement of Intelligence Factors
TLDR
A scale for measuring the complexity of an instance of a problem, depending on the descriptional complexity (Levin LT variant) of the ‘explanation’ of the answer to the problem is developed.
Measuring universal intelligence: Towards an anytime intelligence test
Beyond the Turing Test
TLDR
The idea of using the Turing Test as apractical test of intelligence should be surpassed, and substituted by computational and factorial tests of different cognitive abilities, a more useful approach for artificial intelligence progress and for many other intriguing questions that present themselves beyond theTuring Test.
On More Realistic Environment Distributions for Defining, Evaluating and Developing Intelligence
TLDR
This paper agrees with previous criticisms that a universal distribution using a reference universal Turing machine (UTM) over tasks, environments, etc., is perhaps amuch too general and proposes the notion of Darwin-Wallace distribution for environments, which is inspired by biological evolution, artificial life and evolutionary computation.
IQ tests are not for machines, yet
Human Cognitive Abilities-a sur-vey of factor-analytic studies
Preface Part I. Introduction to the Survey: 1. The study of cognitive abilities 2. Historical foundations of the study of cognitive abilities 3. Survey and analysis of correlational and…
Psychometric artificial intelligence
TLDR
The paper in the present issue of JETAI can be plausibly viewed as resurrecting this approach to machine intelligence, in the form of what is called Psychometric AI, or just PAI (rhymes with ‘ ’).
Universal Intelligence: A Definition of Machine Intelligence
TLDR
A number of well known informal definitions of human intelligence are taken, and mathematically formalised to produce a general measure of intelligence for arbitrary machines that formally captures the concept of machine intelligence in the broadest reasonable sense.
An Approximation of the Universal Intelligence Measure
TLDR
This paper studies the practical issues involved in developing a real-world UIM-based performance metric and develops a prototype implementation which is used to evaluate a number of different artificial agents.
...
1
2
3
4
5
...