Learning task-dependent distributed representations by backpropagation through structure

  title={Learning task-dependent distributed representations by backpropagation through structure},
  author={Christoph Goller and Andreas K{\"u}chler},
  journal={Proceedings of International Conference on Neural Networks (ICNN'96)},
  pages={347-352 vol.1}
  • C. GollerA. Küchler
  • Published 3 June 1996
  • Computer Science
  • Proceedings of International Conference on Neural Networks (ICNN'96)
While neural networks are very successfully applied to the processing of fixed-length vectors and variable-length sequences, the current state of the art does not allow the efficient processing of structured objects of arbitrary shape (like logical terms, trees or graphs. [] Key Method The major difference of our approach compared to others is that the structure-representations are exclusively tuned for the intended inference task.

Figures and Tables from this paper

Inductive Learning in Symbolic Domains Using Structure-Driven Recurrent Neural Networks

A connectionist architecture together with a novel supervised learning scheme which is capable of solving inductive inference tasks on complex symbolic structures of arbitrary size and first results from experiments with inductive learning tasks consisting in the classification of logical terms are given.

Comparing Structures Using a Hopfield-Style Neural Network

The former approaches of structural matching and constraint relaxation by spreading activation in neural networks and the method of solving optimization tasks using Hopfield-style nets are combined.

Learning Efficiently with Neural Networks: A Theoretical Comparison between Structured and Flat Representations

The message of this paper is that, whenever structured representations are available, they should be preferred to "flat" (array based) representations because they are likely to simplify learning in terms of time complexity.

A general framework for adaptive processing of data structures

The framework described in this paper is an attempt to unify adaptive models like artificial neural nets and belief nets for the problem of processing structured information, where relations between data variables are expressed by directed acyclic graphs, where both numerical and categorical values coexist.

The loading problem for recursive neural networks

Neural Representation Learning in Linguistic Structured Prediction

This thesis argues for the importance of modeling discrete structure in language, even when learning continuous representations, and proposes dynamic recurrent acyclic graphical neural networks (DRAGNN), a modular neural architecture that generalizes the encoder/decoder concept to include explicit linguistic structures.

Selective Training: A Strategy for Fast Backpropagation on Sentence Embeddings

This work presents a method to reduce training time substantially by selecting training instances that provide relevant information for training, based on the similarity of the learned representations over input instances, thus allowing for learning a non-trivial weighting scheme from multi-dimensional representations.

Recursive Neural Networks Applied to Discourse Representation Theory

A novel technique is introduced, combining Discourse Representation Theory (DRT) with Recursive Neural Networks (RNN) in order to yield a neural model capable to discover properties and relationships among constituents of a knowledge-base expressed by natural language sentences.

Neural Networks for Processing Data Structures

  • A. Sperduti
  • Computer Science
    Summer School on Neural Networks
  • 1997
It is not difficult to figure out how to extract tree automata from a neural network for structures, and this would allow the above scheme to work on the other side around, with a neural module which is driven by a symbolic subsystem.



Recursive Distributed Representations

Learning Recursive Distributed Representations for Holistic Computation

Two possible forms of holistic inference, transformational inference and confluent inference, are identified and compared and a dual-ported RAAM architecture based on Pollack's Recursive Auto-Associative Memory is implemented and demonstrated in the domain of Natural Language translation.

Distributed representations for terms in hybrid reasoning systems

The intended applications of the approach are hybrid (symbolic/connectionist) systems, where the connectionist part has to solve logic-oriented inductive learning tasks similar to the term-classiication problems used in the experiments.

A Connectionist Parser with Recursive Sentence Structure and Lexical Disambiguation

XERIC networks, presented here, are distributed representation connectionist parsers which can analyze and represent syntactically varied sentences, including ones with recursive phrase structure constructs.

Backpropagation Through Time: What It Does and How to Do It

  • P. Werbos
  • Computer Science, Mathematics
    Proc. IEEE
  • 1990
This paper first reviews basic backpropagation, a simple method which is now being widely used in areas like pattern recognition and fault diagnosis, and describes further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations or true recurrent networks, and other practical issues which arise with this method.

Exploring the Symbolic/Subsymbolic Continuum: A case study of RAAM

It is di cult to clearly de ne the symbolic and subsymbolic paradigms; each is usually described by its tendencies rather than any one de nitive property. Symbolic processing is generally

Encoding Labeled Graphs by Labeling RAAM

The Labeling RAAM (LRAAM), an extension to the RAAM by Pollack, can encode labeled graphs with cycles by representing pointers explicitly by transforming the encoder network of the LRAAM into an analog Hopfield network with hidden units.

Structure Sensitivity in Connectionist Models

Annotation: Published in The Proceedings of the 1993 Connectionist Models Summer School, (Eds) Mozer et al., Lawrence Erlbaum, 1993.

Tree matching with recursive distributed representations