• Corpus ID: 49665167

Measuring abstract reasoning in neural networks

@article{Santoro2018MeasuringAR,
  title={Measuring abstract reasoning in neural networks},
  author={Adam Santoro and Felix Hill and David G. T. Barrett and Ari S. Morcos and Timothy P. Lillicrap},
  journal={ArXiv},
  year={2018},
  volume={abs/1807.04225}
}
Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test… 
Abstract Reasoning with Distracting Features
TLDR
This paper proposes feature robust abstract reasoning (FRAR) model, which consists of a reinforcement learning based teacher network to determine the sequence of training and a student network for predictions that is able to beat the state-of-the-art models.
What Can Neural Networks Reason About?
TLDR
This framework offers an explanation for the empirical success of popular reasoning models, and suggests their limitations, and unify seemingly different reasoning tasks via the lens of a powerful algorithmic paradigm, dynamic programming (DP).
Learning by Abstraction: The Neural State Machine
TLDR
The Neural State Machine is introduced, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning, by transforming both the visual and linguistic modalities into semantic concept-based representations, thereby achieving enhanced transparency and modularity.
Neural Theorem Provers Do Not Learn Rules Without Exploration
TLDR
This work creates synthetic logical datasets with injected relationships, which can be generated on-the-fly, to test neural-based relation learning algorithms including NTP, and alters the NTP algorithm to increase exploration, which sharply improves performance.
EXTRAPOLATABLE RELATIONAL REASONING WITH COMPARATORS IN LOW-DIMENSIONAL MANIFOLDS
  • 2020
While modern deep neural architectures generalise well when test data is sampled from the same distribution as training data, they fail badly for cases when the test data distribution differs from
Learning to Make Analogies by Contrasting Abstract Relational Structure
TLDR
This work studies how analogical reasoning can be induced in neural networks that learn to perceive and reason about raw visual data and finds that the critical factor for inducing such a capacity is not an elaborate architecture, but rather, careful attention to the choice of data and the manner in which it is presented to the model.
Analysing Mathematical Reasoning Abilities of Neural Models
TLDR
This paper conducts a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and finds notable differences in their ability to resolve mathematical problems and generalize their knowledge.
Attention on Abstract Visual Reasoning
TLDR
This work proposes a hybrid network architecture, grounded on self-attention and relational reasoning, and calls this new model Attention Relation Network (ARNe), which combines features from the recently introduced Transformer and the Wild RelATION Network (WReN).
Test Harder than You Train: Probing with Extrapolation Splits
Previous work on probing word representations for linguistic knowledge has focused on interpolation tasks. In this paper, we instead analyse probes in an extrapolation setting, where the inputs at
Generalisable Relational Reasoning With Comparators in Low-Dimensional Manifolds
TLDR
This paper proposes a neuroscience-inspired inductive-biased module that can be readily amalgamated with current neural network architectures to improve out-of-distribution (o.o.d) generalisation performance on relational reasoning tasks, and shows that neural nets with this inductive bias achieve considerably better o.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 34 REFERENCES
A simple neural network module for relational reasoning
TLDR
This work shows how a deep learning architecture equipped with an RN module can implicitly discover and learn to reason about entities and their relations.
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
TLDR
This work presents a diagnostic dataset that tests a range of visual reasoning abilities and uses this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.
Towards Deep Symbolic Reinforcement Learning
TLDR
It is shown that the resulting system -- though just a prototype -- learns effectively, and, by acquiring a set of symbolic rules that are easily comprehensible to humans, dramatically outperforms a conventional, fully neural DRL system on a stochastic variant of the game.
Intriguing properties of neural networks
TLDR
It is found that there is no distinction between individual highlevel units and random linear combinations of high level units, according to various methods of unit analysis, and it is suggested that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks.
Building machines that learn and think for themselves
TLDR
It is believed an approach centered on autonomous learning has the greatest chance of success as the authors scale toward real-world complexity, tackling domains for which ready-made formal models are not available.
SCAN: Learning Abstract Hierarchical Compositional Visual Concepts
TLDR
SCAN (Symbol-Concept Association Network), a new framework for learning concepts in the visual domain capable of multimodal bi-directional inference and traversal and manipulation of the implicit hierarchy of compositional visual concepts through symbolic instructions and learnt logical recombination operations, is described.
Understanding deep learning requires rethinking generalization
TLDR
These experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data, and confirm that simple depth two neural networks already have perfect finite sample expressivity.
Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks
TLDR
This paper introduces the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences, and tests the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods.
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
TLDR
Experimental results show that indirect supervision of program learning via answer rationales is a promising strategy for inducing arithmetic programs.
My Computer Is an Honor Student - but How Intelligent Is It? Standardized Tests as a Measure of AI
TLDR
It is argued that machine performance on standardized tests should be a key component of any new measure of AI, because attaining a high level of performance requires solving significant AI problems involving language understanding and world modeling - critical skills for any machine that lays claim to intelligence.
...
1
2
3
4
...