• Corpus ID: 6211222

Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data

@article{Mocanu2016OnlineCD,
  title={Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data},
  author={Decebal Constantin Mocanu and Maria Torres Vega and Eric Eaton and Peter Stone and Antonio Liotta},
  journal={ArXiv},
  year={2016},
  volume={abs/1610.05555}
}
Conceived in the early 1990s, Experience Replay (ER) has been shown to be a successful mechanism to allow online learning algorithms to reuse past experiences. Traditionally, ER can be applied to all machine learning paradigms (i.e., unsupervised, supervised, and reinforcement learning). Recently, ER has contributed to improving the performance of deep reinforcement learning. Yet, its application to many practical settings is still limited by the memory requirements of ER, necessary to… 

Figures and Tables from this paper

Continual Learning with Deep Generative Replay
TLDR
The Deep Generative Replay is proposed, a novel framework with a cooperative dual model architecture consisting of a deep generative model ("generator") and a task solving model ("solver"), with only these two models, training data for previous tasks can easily be sampled and interleaved with those for a new task.
Deep Generative Dual Memory Network for Continual Learning
TLDR
This model consists of a dual memory architecture to emulate the complementary learning systems (hippocampus and the neocortex) in the human brain, and maintains a consolidated long-term memory via generative replay of past experiences.
Learning Invariant Representation for Continual Learning
TLDR
A new pseudo-rehearsalbased method is proposed, named learning Invariant Representation for Continual Learning (IRCL), in which class-invariant representation is disentangled from a conditional generative model and jointly used with class-specific representation to learn the sequential tasks.
Addressing the Stability-Plasticity Dilemma via Knowledge-Aware Continual Learning
TLDR
KAN is a new task-specific components method based on dynamic sparse training that introduces reusability and selective transfer of past knowledge in this class of methods, and selectively reuses relevant knowledge while addressing class ambiguity, preserves old knowledge, and utilizes the model capacity efficiently.
Self-Attention Meta-Learner for Continual Learning
TLDR
This paper proposes a new method, named Self-Attention MetaLearner (SAM) which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting, and empirically shows that it can achieve a better performance than several state-of-the-art methods by building on top of the selected representation learned by SAM.
Generative Memory for Lifelong Reinforcement Learning
TLDR
This paper proposes the use ofGenerative memory that can be recalled in batch samples to train a multi-task agent in a pseudo-rehearsal manner and shows results motivating the need for task-agnostic separation of latent space for the generative memory to address issues of catastrophic forgetting in lifelong learning.
One-Shot Learning using Mixture of Variational Autoencoders: a Generalization Learning approach
TLDR
This paper proposes a new one-shot learning method, dubbed MoVAE (Mixture of Variational AutoEncoders), to perform classification, and demonstrates empirically the efficiency of the proposed approach on three datasets, i.e. the handwritten digits, fashion products, and handwritten characters, that outperforms state-of-the-art one- shot learning algorithms.
One-shot learning using Mixture of Variational Autoencoders : A generalization learning approach
TLDR
This paper proposes a new one-shot learning method, dubbed MoVAE (Mixture of Variational AutoEncoders), to perform classification, and demonstrates empirically the efficiency of the proposed approach on three datasets, i.e. the handwritten digits, fashion products, and handwritten characters, that outperforms state-of-the-art oneshot learning algorithms.
One-shot learning using Mixture of Variational Autoencoders
TLDR
This paper proposes a new one-shot learning method, dubbed MoVAE (Mixture of Variational AutoEncoders), to perform classification, and demonstrates empirically the efficiency of the proposed approach on three datasets, i.e. the handwritten digits, fashion products, and handwritten characters, that outperforms state-of-the-art oneshot learning algorithms.
Memory Augmented Neural Network with Gaussian Embeddings for One-Shot Learning
TLDR
A memory augmented network with Gaussian embeddings is constructed and since the resulting model is generative, image reconstruction results on Omniglot and LFW-a datasets are provided.
...
1
2
...

References

SHOWING 1-10 OF 39 REFERENCES
Prioritized Experience Replay
TLDR
A framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently, in Deep Q-Networks, a reinforcement learning algorithm that achieved human-level performance across many Atari games.
Batch reinforcement learning in a complex domain
TLDR
This paper compares and contrast batch reinforcement learning algorithms with on-line algorithms based on their empirical performance in a complex, continuous, noisy, multiagent domain, namely RoboCup soccer Keepaway.
Using fast weights to improve persistent contrastive divergence
TLDR
It is shown that the weight updates force the Markov chain to mix fast, and using this insight, an even faster mixing chain is developed that uses an auxiliary set of "fast weights" to implement a temporary overlay on the energy landscape.
Human-level control through deep reinforcement learning
TLDR
This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
MADE: Masked Autoencoder for Distribution Estimation
TLDR
This work introduces a simple modification for autoencoder neural networks that yields powerful generative models and proves that this approach is competitive with state-of-the-art tractable distribution estimators.
Learning Deep Architectures for AI
TLDR
The motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer modelssuch as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks are discussed.
Experience Replay for Real-Time Reinforcement Learning Control
TLDR
This paper evaluates ER RL on real-time control experiments that involve a pendulum swing-up problem and the vision-based control of a goalkeeper robot, and develops a general ER framework that can be combined with essentially any incremental RL technique, and instantiate this framework for the approximate Q-learning and SARSA algorithms.
Parallel Tempering for Training of Restricted Boltzmann Machines
TLDR
This work explores the use of tempered Markov Chain Monte-Carlo for sampling in RBMs and finds both through visualization of samples and measures of likelihood on a toy dataset that it helps both sampling and learning.
A topological insight into restricted Boltzmann machines
TLDR
This work shows that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology, and shows that, for a fixed number of weights, the proposed sparse models achieve better generative capabilities than standard fully connected RBs and GRBMs, at no additional computational costs.
An automated measure of MDP similarity for transfer in reinforcement learning
TLDR
A data-driven automated similarity measure for Markov Decision Processes, based on the reconstruction error of a restricted Boltzmann machine that attempts to model the behavioral dynamics of the two MDPs being compared, which can be used to identify similar source tasks for transfer learning.
...
1
2
3
4
...