Learning and development in neural networks: the importance of starting small

  title={Learning and development in neural networks: the importance of starting small},
  author={Jeffrey L. Elman},
  • J. Elman
  • Published 1993
  • Medicine, Psychology
  • Cognition
It is a striking fact that in humans the greatest learning occurs precisely at that point in time--childhood--when the most dramatic maturational changes also occur. This report describes possible synergistic interactions between maturational change and the ability to learn a complex domain (language), as investigated in connectionist networks. The networks are trained to process complex sentences involving relative clauses, number agreement, and several types of verb argument structure… Expand
Learning and development in neural networks – the importance of prior experience
A simple recurrent network, equipped only with the assumption that it should predict what comes next, is described, which models the data without distinguishing between familiarization and discrimination. Expand
Neural network processing of natural language: I. Sensitivity to serial, temporal and abstract structure of language in the infant
Well before their first birthday, babies can acquire knowledge of serial order relations (Saffran et al., 1996a), as well as knowledge of more abstract rule-based structural relations (Marcus et al.,Expand
Simple Recurrent Networks and Natural Language: How Important is Starting Small?
Evidence is reported that starting with simplified inputs is not necessary in training recurrent networks to learn pseudo-natural languages and it is suggested that the structure of natural language can be learned without special teaching methods or limited cognitive resources. Expand
Language acquisition in the absence of explicit negative evidence: how important is starting small?
It is commonly assumed that innate linguistic constraints are necessary to learn a natural language, based on the apparent lack of explicit negative evidence provided to children and on Gold's proofExpand
The sensitive period for language acquisition: The role of age related differences in cognitive and neural function
The aim of this research is to better understand why children consistently surpass adults in their ultimate attainment of language--the sensitive period for language acquisition. I propose the NestedExpand
Language Learning in Infancy
Publisher Summary Language is a complex, specialized skill, which develops in the child spontaneously, without conscious effort or formal instruction, is deployed without awareness of its underlyingExpand
The evolution of incremental learning: language, development and critical periods
In this article 1 , we show how the existence of critical periods follows from the action of natural selection on genomes in which incremental growth can be tuned to chronological age (maturation) orExpand
The Role of Prior Experience in Language Acquisition
Findings suggest that learners, and the structure they can acquire, change as a function of experience, and that learners exposed to an artificial language recognize its abstract structural regularities when instantiated in a novel vocabulary. Expand
Connectionist approaches to language learning
Abstract In the past twenty years the connectionist approach to language development and learning has emerged as an alternative to traditional linguistic theories. This article introduces theExpand
Functional Innateness: explaining the critical period for language acquisition
In recent years, several explanations have been offered for the critical period in language acquisition, itself, a priori a somewhat surprising phenomenon. Two such explanations are considered here.Expand


Maturational Constraints on Language Learning
  • E. Newport
  • Psychology, Computer Science
  • Cogn. Sci.
  • 1990
This paper suggests that there are constraints on learning required to explain the acquisition of language, in particular, mului ultonol constraints, and suggests that language learning abilities decline because of the expansion of nonlinguisftc cognitive abilities. Expand
Parallel distributed processing models and metaphors for language and development
This dissertation focuses on one type of PDP model, networks of simple processing units trained by the back propagation algorithm to learn input-output relationships, and the potential of P DP models to capture linguistic regularities. Expand
Constraints on learning and their role in language acquisition: Studies of the acquisition of American sign language
Abstract The general question raised here is why the young child is superior to older children and adults at language acquisition, while at the same time inferior to them in many other cognitiveExpand
Parallel Distributed Processing: Implications for Cognition and Development
The application of the connectionist framework to problems of cognitive development is considered, and a network that learns to anticipate which side of a balance beam will go down is illustrated, based on the number of weights on each side of the fulcrum and their distance from the Fulcrum. Expand
Finding Structure in Time
A proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory and suggests a method for representing lexical categories and the type/token distinction is developed. Expand
Distributed Representations, Simple Recurrent Networks, and Grammatical Structure
  • J. Elman
  • Mathematics, Computer Science
  • Mach. Learn.
  • 1991
Using a prediction task, a simple recurrent network is trained on multiclausal sentences which contain multiply-embedded relative clauses and principal component analysis of the hidden unit activation patterns reveals that the network solves the task by developing complex distributed representations which encode the relevant grammatical relations and hierarchical constituent structure. Expand
Learning Subsequential Structure in Simple Recurrent Networks
A network architecture introduced by Elman (1988) for predicting successive elements of a sequence using the pattern of activation over a set of hidden units to be illustrated with cluster analyses performed at different points during training. Expand
Limitations on input as a basis for neural organization and perceptual development: a preliminary theoretical statement.
According to this view, limitations, particularly of the sensory systems, produce adaptive advantages for infants by facilitating perceptual organization and result in relative independence among emerging systems, thereby reducing competition which helps regulate subsequent neurogenesis and functioning. Expand
The development of memory in Children
Charting the development of several different facets of memory, "The Development of Memory in Children" shows how developmental changes in memory relate to more general cognitive changes. Robert KailExpand
Encoding sequential structure: experience with the real-time recurrent learning algorithm
It is shown that recurrent nets trained with the RTRL (real-time recurrent learning) algorithm are able to learn tasks that Elman nets appear unable to learn. Moreover, they learn a more stringentExpand