• Corpus ID: 15299054

Neural Turing Machines

@article{Graves2014NeuralTM,
  title={Neural Turing Machines},
  author={Alex Graves and Greg Wayne and Ivo Danihelka},
  journal={ArXiv},
  year={2014},
  volume={abs/1410.5401}
}
We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. [] Key Result Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.

Neural networks with external dynamic memory to solve algorithmic problems

TLDR
Preliminary results show that Differentiable Neural Computers can infer simple algorithms like Neural Turing Machines but in a more efficient and efficacy way, and also has the capability of generalizing results without retraining the model.

Evolving Neural Turing Machines!

TLDR
Preliminary results suggest that this setup can greatly simplify the neural model, generalizes better, and does not require accessing the entire memory content at each time-step.

Learning Operations on a Stack with Neural Turing Machines

TLDR
It is shown that not only does the NTM emulate a stack with its heads and learn an algorithm to recognize such words, but it is also capable of strongly generalizing to much longer sequences.

A review on Neural Turing Machine

TLDR
The attempt is made here to run a systematic review on Neural Turing Machine in terms of concepts, structure, variety of versions, implemented tasks, comparisons, etc.

Neural Turing Machines: Convergence of Copy Tasks

The architecture of neural Turing machines is differentiable end to end and is trainable with gradient descent methods. Due to their large unfolded depth Neural Turing Machines are hard to train and

Implementing Neural Turing Machines

TLDR
This paper finds that the choice of memory contents initialization scheme is crucial in successfully implementing a NTM, and finds networks with memory contents initialized to small constant values converge on average 2 times faster than the next best memory contents initialize scheme.

Neural Structured Turing Machine

TLDR
This work takes structured information into consideration and proposes a Neural Structured Turing Machine (NSTM), leading to a novel perspective for improving differential external memory based mechanism, e.g., NTM.

Continual Learning through Evolvable Neural Turing Machines

TLDR
This paper shows that the recently proposed Evolving Neural Turing Machine (ENTM) approach is able to perform one-shot learning in a reinforcement learning task without catastrophic forgetting of previously stored associations.

Structured Memory for Neural Turing Machines

TLDR
This paper proves in experiments that two of the proposed structured-memory NTMs could lead to better convergence, in term of speed and prediction accuracy on copy task and associative recall task as in (Graves et al. 2014).

Neural Stored-program Memory

TLDR
A new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures is introduced, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus resemble the Universal Turing Machine.
...

References

SHOWING 1-10 OF 44 REFERENCES

On the Computational Power of Neural Nets

TLDR
It is proved that one may simulate all Turing Machines by rational nets in linear time, and there is a net made up of about 1,000 processors which computes a universal partial-recursive function.

Neural networks and physical systems with emergent collective computational abilities.

  • J. Hopfield
  • Computer Science
    Proceedings of the National Academy of Sciences of the United States of America
  • 1982
TLDR
A model of a system having a large number of simple equivalent components, based on aspects of neurobiology but readily adapted to integrated circuits, produces a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size.

Recursive Distributed Representations

Simple Substrates for Complex Cognition

  • P. Dayan
  • Computer Science
    Front. Neurosci.
  • 2008
TLDR
A recent suggestion for a uniform architecture for habitual and rule-based execution is reviewed, some of the habitual mechanisms that underpin the use of rules are discussed, and a statistical relationship between rules and habits is considered.

The Algebraic Mind: Integrating Connectionism and Cognitive Science

TLDR
Gary Marcus outlines a variety of ways in which neural systems could be organized so as to manipulate symbols, and he shows why such systems are more likely to provide an adequate substrate for language and cognition than neural systems that are inconsistent with the manipulation of symbols.

Continuous attractors and oculomotor control

BoltzCONS: Dynamic Symbol Structures in a Connectionist Network

Cellular basis of working memory

Banishing the homunculus: Making working memory work

Generating Text with Recurrent Neural Networks

TLDR
The power of RNNs trained with the new Hessian-Free optimizer by applying them to character-level language modeling tasks is demonstrated, and a new RNN variant that uses multiplicative connections which allow the current input character to determine the transition matrix from one hidden state vector to the next is introduced.