Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks

@article{Giles1992LearningAE,
  title={Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks},
  author={C. Lee Giles and Clifford B. Miller and Dong Chen and Hsing-Hen Chen and Guo-Zheng Sun and Yee-Chun Lee},
  journal={Neural Computation},
  year={1992},
  volume={4},
  pages={393-405}
}
We show that a recurrent, second-order neural network using a real-time, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples. We present simulations that show the effect of initial conditions, training set size and order, and neural network architecture. All simulations were performed with random initial weight strengths and usually converge after approximately a hundred epochs of training. We discuss a quantization… 
View on MIT Press
watermark.silverchair.com
490 Citations
Highly Influential Citations
44
Background Citations
157
Methods Citations
110
Results Citations
11

490 Citations

First-Order Recurrent Neural Networks and Deterministic Finite State Automata
TLDR
The correspondence between first-order recurrent neural networks and deterministic finite state automata is examined in detail, showing two major stages in the learning process and a measure based on clustering that is correlated to the stability of the networks.
Constrained Second-Order Recurrent Networks for Finite-State Automata Induction
TLDR
A simple modification to the standard error function for second-order dynamical recurrent networks is suggested which encourages these networks to assume natural FSA encodings when trained using gradient descent and provides a simple method for guaranteeing the stability of the network for arbitrarily long sequences.
Discrete recurrent neural networks for grammatical inference
TLDR
A novel neural architecture for learning deterministic context-free grammars, or equivalently, deterministic pushdown automata is described, and a composite error function is described to handle the different situations encountered in learning.
Rule Revision With Recurrent Neural Networks
TLDR
The results from training a recurrent neural network to recognize a known non-trivial, randomly-generated regular grammar show that not only do the networks preserve correct rules but that they are able to correct through training inserted rules which were initially incorrect.
Extracting and Learning an Unknown Grammar with Recurrent Neural Networks
Simple second-order recurrent networks are shown to readily learn small known regular grammars when trained with positive and negative strings examples. We show that similar methods are appropriate…
Dynamic Adaptation of Recurrent Neural Network Architectures Guided by Symbolic Knowledge
TLDR
This work proposes a novel method for dynamically adapting the architecture of recurrent neural networks trained to behave like determinis-tic nite-state automata (DFAs), which relies on the continuous extraction and insertion of symbolic knowledge in the form of DFAs.
Representation and Recognition of Regular Grammars by Means of Second-Order Recurrent Neural Networks
TLDR
This work addresses and solves a basic problem, which is, how to build a neural net recognizer for a given regular language specified by a deterministic finite-state automaton, and employs a second-order recurrent network model, which allows to formulate the problem as one of solving a linear system of equations.
Learning a class of large finite state machines with a recurrent neural network
Extraction of rules from discrete-time recurrent neural networks
Learning Finite State Machines With Self-Clustering Recurrent Networks
TLDR
This paper proposes a new method to force a recurrent neural network to learn stable states by introducing discretization into the network and using a pseudo-gradient learning rule to perform training, which has similar capabilities in learning finite state automata as the original network, but without the instability problem.
...
...

References

SHOWING 1-10 OF 31 REFERENCES
Discrete recurrent neural networks for grammatical inference
TLDR
A novel neural architecture for learning deterministic context-free grammars, or equivalently, deterministic pushdown automata is described, and a composite error function is described to handle the different situations encountered in learning.
Learning Finite State Machines With Self-Clustering Recurrent Networks
TLDR
This paper proposes a new method to force a recurrent neural network to learn stable states by introducing discretization into the network and using a pseudo-gradient learning rule to perform training, which has similar capabilities in learning finite state automata as the original network, but without the instability problem.
Finite State Automata and Simple Recurrent Networks
TLDR
A network architecture introduced by Elman (1988) for predicting successive elements of a sequence and shows that long distance sequential contingencies can be encoded by the network even if only subtle statistical properties of embedded strings depend on the early information.
Heuristics for the extraction of rules from discrete-time recurrent neural networks
TLDR
Empirical evidence that there exists a correlation between the generalization performance of recurrent neural networks for regular language recognition and the rules that can be extracted from a neural network is presented and a heuristic that makes it possible to extract good rules from trained networks is given.
Higher Order Recurrent Networks and Grammatical Inference
TLDR
A higher order single layer recursive network easily learns to simulate a deterministic finite state machine and recognize regular grammars and can be interpreted as a neural net pushdown automata.
Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks
TLDR
This paper discusses various issues: how rules are inserted into recurrent networks, how they affect training and generalization, and how those rules can be checked and corrected.
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal…
Diversity-based inference of finite automata
  • R. Rivest, R. Schapire
  • Computer Science
    28th Annual Symposium on Foundations of Computer Science (sfcs 1987)
  • 1987
TLDR
A new procedure for inferring the structure of a finitestate automaton (FSA) from its input/output behavior, using access to the automaton to perform experiments, based on the notion of equivalence between testa.
Induction of Finite-State Languages Using Second-Order Recurrent Networks
TLDR
Second-order recurrent networks that recognize simple finite state languages over {0,1}* are induced from positive and negative examples to obtain solutions that correctly recognize strings of arbitrary length.
Complexity of Automaton Identification from Given Data
  • E. M. Gold
  • Computer Science, Mathematics
    Inf. Control.
  • 1978
...
...