Share This Author
On the Practical Computational Power of Finite Precision RNNs for Language Recognition
It is shown that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU.
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples
We present a novel algorithm that uses exact learning and abstraction to extract a deterministic finite automaton describing the state dynamics of a given trained RNN. We do this using Angluin's L*…
Learning Deterministic Weighted Automata with Queries and Counterexamples
The algorithm is a variant of the exact-learning algorithm L*, adapted to work in a probabilistic setting under noise, and uses the use of conditional probabilities when making observations on the model, and the introduction of a variation tolerance when comparing observations.
Thinking Like Transformers
This paper proposes a computational model for the transformer-encoder in the form of a programming language, the Restricted Access Sequence Processing Language (RASP), and provides RASP programs for histograms, sorting, and Dyck-languages.
A Formal Hierarchy of RNN Architectures
- William Cooper Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, Eran Yahav
- Computer ScienceACL
- 1 April 2020
It is hypothesized that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy, and empirical results to support this conjecture are provided.
Synthesizing Context-free Grammars from Recurrent Neural Networks
An algorithm for extracting a subclass of the context free grammars (CFGs) from a trained recurrent neural network (RNN) is presented, and how the PRS may be converted into a CFG, enabling a familiar and useful presentation of the learned language.