Constructing deterministic finite-state automata in recurrent neural networks

@article{Omlin1996ConstructingDF,
  title={Constructing deterministic finite-state automata in recurrent neural networks},
  author={Christian W. Omlin and C. Lee Giles},
  journal={J. ACM},
  year={1996},
  volume={43},
  pages={937-972}
}
Recurrent neural networks that are <italic>trained</italic> to behave like deterministic finite-state automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidel discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can <italic>construct</italic> second-order… 
View on ACM
drum.lib.umd.edu
184 Citations
Highly Influential Citations
11
Background Citations
71
Methods Citations
40
Results Citations
7

184 Citations

Citation Type

Citation Type

Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants
We propose an algorithm for encoding deterministic finite-state automata (DFAs) in second-order recurrent neural networks with sigmoidal discriminant function and we prove that the languages accepted…
Efficient encodings of finite automata in discrete-time recurrent neural networks ∗
TLDR
This paper shows that more efficient sigmoid DTRNN encodings exist for a subclass of deterministic finite automata (DFA), namely, when the size of an equivalent nondeterministic finite Automata (NFA) is smaller, because n-state NFA may directly be encoded in D TRNN with a Θ(n) units.
The Neural Network Pushdown Automaton: Architecture, Dynamics and Training
TLDR
A new model, a neural network pushdown automaton (NNPDA), which is a hybrid system that couples a recurrent network to an external stack memory and should be capable of learning and recognizing some class of context-free grammars.
On the Computational Power of Recurrent Neural Networks for Structures
Simple Strategies to Encode Tree Automata in Sigmoid Recursive Neural Networks
TLDR
This work explores the application of a recent result that any threshold linear unit operating on binary inputs can be implemented in an analog unit using a continuous activation function and bounded real inputs, and presents an alternative scheme for one-hot encoding of the input that yields smaller weight values, and therefore works at a lower saturation level.
Representing Formal Languages: A Comparison Between Finite Automata and Recurrent Neural Networks
TLDR
A strong structural relationship between internal representations used by RNNs and finite automata is suggested, and this helps explain the well-known ability of RNN's to recognize formal grammatical structure.
Stable Encoding of Finite-State Machines in Discrete-Time Recurrent Neural Nets with Sigmoid Units
TLDR
This article describes a simple strategy to devise stable encodings of finite-state machines in computationally capable discrete-time recurrent neural architectures with sigmoid units and gives a detailed presentation on how this strategy may be applied to encode a general class of infinite- state machines in a variety of commonly used first- and second-order recurrent neural networks.
Fuzzy finite-state automata can be deterministically encoded into recurrent neural networks
TLDR
This work proposes an algorithm that constructs an augmented recurrent neural network that encodes a FFA and recognizes a given fuzzy regular language with arbitrary accuracy and examines how the networks' performance varies as a function of synaptic weight strengths.
A Neural State Pushdown Automata
TLDR
A “neural state” pushdown automaton (NSPDA), which consists of a discrete stack instead of an continuous one and is coupled to a neural network state machine, and empirically shows its effectiveness in recognizing various context-free grammars (CFGs).
Equivalence in knowledge representation: automata, recurrent neural networks, and dynamical fuzzy systems
TLDR
Various knowledge equivalence representations between neural and fuzzy systems and models of automata are proved and the stability of fuzzy finite state dynamics of the constructed neural networks for finite values of network weight is proved.
...
...

References

SHOWING 1-10 OF 58 REFERENCES
Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants
We propose an algorithm for encoding deterministic finite-state automata (DFAs) in second-order recurrent neural networks with sigmoidal discriminant function and we prove that the languages accepted…
Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks
TLDR
It is shown that a recurrent, second-order neural network using a real-time, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples, and many of the neural net state machines are dynamically stable, that is, they correctly classify many long unseen strings.
Second-order recurrent neural networks for grammatical inference
TLDR
It is shown that a recurrent, second-order neural network using a real-time, feedforward training algorithm readily learns to infer regular grammars from positive and negative string training samples and many of the neural net state machines are dynamically stable and correctly classify long unseen strings.
Learning Finite State Machines With Self-Clustering Recurrent Networks
TLDR
This paper proposes a new method to force a recurrent neural network to learn stable states by introducing discretization into the network and using a pseudo-gradient learning rule to perform training, which has similar capabilities in learning finite state automata as the original network, but without the instability problem.
Rule refinement with recurrent neural networks
TLDR
The results from training a recurrent neural network to recognize a known nontrivial randomly generated regular grammar show that not only do the networks preserve correct prior knowledge, but they are able to correct through training inserted prior knowledge which was wrong.
Rule Revision With Recurrent Neural Networks
TLDR
The results from training a recurrent neural network to recognize a known non-trivial, randomly-generated regular grammar show that not only do the networks preserve correct rules but that they are able to correct through training inserted rules which were initially incorrect.
The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction
TLDR
It is shown that an RNN performing a finite state computation must organize its state space to mimic the states in the minimal deterministic finite state machine that can perform that computation, and a precise description of the attractor structure of such systems is given.
Efficient simulation of finite automata by neural nets
TLDR
Constraints on the local structure of the network give, by a counting argument and a construction, lower and upper bounds for K(m) that are both linear in m.
Inserting rules into recurrent neural networks
  • C. L. Giles, C. Omlin
  • Computer Science
    Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop
  • 1992
TLDR
Simulations show that training recurrent networks with different amounts of partial knowledge to recognize simple grammers improves the training times by orders of magnitude, even when only a small fraction of all transitions are inserted as rules.
Bounds on the complexity of recurrent neural network implementations of finite state machines
...
...