• Corpus ID: 15299054

Neural Turing Machines

@article{Graves2014NeuralTM,
  title={Neural Turing Machines},
  author={Alex Graves and Greg Wayne and Ivo Danihelka},
  journal={ArXiv},
  year={2014},
  volume={abs/1410.5401}
}
We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. [] Key Result Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.
Neural networks with external dynamic memory to solve algorithmic problems
TLDR
Preliminary results show that Differentiable Neural Computers can infer simple algorithms like Neural Turing Machines but in a more efficient and efficacy way, and also has the capability of generalizing results without retraining the model.
Evolving Neural Turing Machines!
TLDR
Preliminary results suggest that this setup can greatly simplify the neural model, generalizes better, and does not require accessing the entire memory content at each time-step.
Learning Operations on a Stack with Neural Turing Machines
TLDR
It is shown that not only does the NTM emulate a stack with its heads and learn an algorithm to recognize such words, but it is also capable of strongly generalizing to much longer sequences.
A review on Neural Turing Machine
TLDR
The attempt is made here to run a systematic review on Neural Turing Machine in terms of concepts, structure, variety of versions, implemented tasks, comparisons, etc.
Neural Turing Machines: Convergence of Copy Tasks
The architecture of neural Turing machines is differentiable end to end and is trainable with gradient descent methods. Due to their large unfolded depth Neural Turing Machines are hard to train and
Turing Computation with Recurrent Artificial Neural Networks
TLDR
This work provides a novel and parsimonious constructive mapping between Turing Machines and Recurrent Artificial Neural Networks, based on recent developments of Nonlinear Dynamical Automata, and provides a framework to directly program the R-ANNs from Turing Machine descriptions, in absence of network training.
Implementing Neural Turing Machines
TLDR
This paper finds that the choice of memory contents initialization scheme is crucial in successfully implementing a NTM, and finds networks with memory contents initialized to small constant values converge on average 2 times faster than the next best memory contents initialize scheme.
Neural Structured Turing Machine
TLDR
This work takes structured information into consideration and proposes a Neural Structured Turing Machine (NSTM), leading to a novel perspective for improving differential external memory based mechanism, e.g., NTM.
Continual Learning through Evolvable Neural Turing Machines
TLDR
This paper shows that the recently proposed Evolving Neural Turing Machine (ENTM) approach is able to perform one-shot learning in a reinforcement learning task without catastrophic forgetting of previously stored associations.
Neural Stored-program Memory
TLDR
A new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures is introduced, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus resemble the Universal Turing Machine.
...
...

References

SHOWING 1-10 OF 44 REFERENCES
On the Computational Power of Neural Nets
TLDR
It is proved that one may simulate all Turing Machines by rational nets in linear time, and there is a net made up of about 1,000 processors which computes a universal partial-recursive function.
Neural networks and physical systems with emergent collective computational abilities.
  • J. Hopfield
  • Computer Science
    Proceedings of the National Academy of Sciences of the United States of America
  • 1982
TLDR
A model of a system having a large number of simple equivalent components, based on aspects of neurobiology but readily adapted to integrated circuits, produces a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size.
Learning Context-free Grammars: Capabilities and Limitations of a Recurrent Neural Network with an External Stack Memory
TLDR
An analog stack is developed which reverts to a discrete stack by quantization of all activations, after the network has learned the transition rules and stack actions, and an enhancement of the network's learning capabilities by providing hints.
Recursive Distributed Representations
Simple Substrates for Complex Cognition
  • P. Dayan
  • Computer Science
    Front. Neurosci.
  • 2008
TLDR
A recent suggestion for a uniform architecture for habitual and rule-based execution is reviewed, some of the habitual mechanisms that underpin the use of rules are discussed, and a statistical relationship between rules and habits is considered.
The Algebraic Mind: Integrating Connectionism and Cognitive Science
TLDR
Gary Marcus outlines a variety of ways in which neural systems could be organized so as to manipulate symbols, and he shows why such systems are more likely to provide an adequate substrate for language and cognition than neural systems that are inconsistent with the manipulation of symbols.
Generating Sequences With Recurrent Neural Networks
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach
Continuous attractors and oculomotor control
BoltzCONS: Dynamic Symbol Structures in a Connectionist Network
A general framework for adaptive processing of data structures
TLDR
The framework described in this paper is an attempt to unify adaptive models like artificial neural nets and belief nets for the problem of processing structured information, where relations between data variables are expressed by directed acyclic graphs, where both numerical and categorical values coexist.
...
...