Many-Layered Learning

  title={Many-Layered Learning},
  author={Paul E. Utgoff and David J. Stracuzzi},
  journal={Neural Computation},
We explore incremental assimilation of new knowledge by sequential learning. Of particular interest is how a network of many knowledge layers can be constructed in an on-line manner, such that the learned units represent building blocks of knowledge that serve to compress the overall representation and facilitate transfer. We motivate the need for many layers of knowledge, and we advocate sequential learning as an avenue for promoting the construction of layered knowledge structures. Finally… 

Cross-Domain Knowledge Transfer Using Structured Representations

A novel learning method, based on neuroevolution, is presented, which uses many-layered, sparsely-connected neural networks in order to learn a structural representation of tasks, and mines frequent sub-graphs to discover sub-networks that are useful for multiple tasks.


The SCALE algorithm is presented as a method for knowledge acquisition and organization, and is used to demonstrate both the computational and training benefits of memory organization.

Memory Organization and Knowledge Transfer

It is proposed that previously acquired knowledge must be organized, and one possible approach is demonstrated to select the appropriate knowledge structures to transfer when many are available.

The Utility of Knowledge Transfer for Noisy Data

Experimental results are given that demonstrate the increased importance of knowledge transfer when learning new concepts from noisy data and exploit the layered nature of DCNN’s to discover more efficient and targeted methods of transfer.

On-Line Cumulative Learning of Hierarchical Sparse -grams

A novel sparse -gram model that, unlike pruned -grams, learns on-line by stochastic search for frequent -tuple patterns, and is an example of a rare combination—unsupervised, on- line, cumulative, structure learning.

The Utility of Knowledge Transfer with Noisy Training Sets

Experimental results are given that demonstrate the increased importance of knowledge transfer when learning new concepts from noisy data and exploit the layered nature of deep convolutional neural nets (DCNN’s) to discover more efficient and targeted methods of transfer.

Learning Deep Architectures for AI

The motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer modelssuch as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks are discussed.

Scaling learning algorithms towards AI

It is argued that deep architectures have the potential to generalize in non-local ways, i.e., beyond immediate neighbors, and that this is crucial in order to make progress on the kind of complex tasks required for artificial intelligence.

Draft: Deep Learning in Neural Networks: An Overview

This historical survey compactly summarises relevant work, much of it from the previous millennium, on deep supervised learning, unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

Hierarchical Learning

A broad model which includes several degrees of freedom which modulate the tradeoff between the effort of the learner and that of the teacher is outlined, as well as a discussion of potential avenues for progress.



Layered Learning

This paper presents layered learning, a hierarchical machine learning paradigm that seamlessly integrates separate learning at each subtask layer, and introduces layered learning in its domain-independent general form.

Using Knowledge to Speed Learning: A Comparison of Knowledge-based Cascade-correlation and Multi-task Learning

This paper reports a comparison of KBCC and MTL for their ability to use knowledge to speed learning and to study of the effects of knowledge relevance on learning speed.

Learning by Discovering Concept Hierarchies

Relational Learning for NLP using Linear Threshold Elements

A coherent view of learning and reasoning with relational representations in the context of natural language processing is described and extensions of this system along the lines suggested by the theory may provide new levels of scalability and functionality.

The Cascade-Correlation Learning Architecture

The Cascade-Correlation architecture has several advantages over existing algorithms: it learns very quickly, the network determines its own size and topology, it retains the structures it has built even if the training set changes, and it requires no back-propagation of error signals through the connections of the network.

A neuroidal architecture for cognitive computation

The main claims are that the basic learning and deduction tasks are provably tractable and tractable learning offers viable approaches to a range of issues that have been previously identified as problematic for artificial intelligence systems that are programmed.

Learning in feedforward layered networks: the tiling algorithm

A new algorithm which builds a feedforward layered network in order to learn any Boolean function of N Boolean units, which is an algorithm for growth of the network, which adds layers, and units inside a layer, at will until convergence.

A reply to Honavar's book review of Neural Network Design and the Complexity of Learning

This work formalizes a notion of learning that characterizes the training of feed-forward networks and introduces a perspective on shallow networks, called the Support Cone Interaction graph, which is helpful in distinguishing tractable from intractable subcases.

Multitask Learning

Prior work on MTL is reviewed, new evidence that MTL in backprop nets discovers task relatedness without the need of supervisory signals is presented, and new results for MTL with k-nearest neighbor and kernel regression are presented.