Transfer of samples in batch reinforcement learning

@inproceedings{Lazaric2008TransferOS,
  title={Transfer of samples in batch reinforcement learning},
  author={Alessandro Lazaric and Marcello Restelli and Andrea Bonarini},
  booktitle={ICML '08},
  year={2008}
}
The main objective of transfer in reinforcement learning is to reduce the complexity of learning the solution of a target task by effectively reusing the knowledge retained from solving a set of source tasks. In this paper, we introduce a novel algorithm that transfers samples (i.e., tuples ⟨s, a, s', r⟩) from source to target tasks. Under the assumption that tasks have similar transition models and reward functions, we propose a method to select samples from the source tasks that are mostly… 
Improving Batch Reinforcement Learning Performance through Transfer of Samples
TLDR
This paper introduces a novel algorithm that selectively transfers samples from source to target tasks and that uses them as input for batch reinforcement-learning algorithms and shows that the proposed approach is effective in reducing the learning complexity.
Importance Weighted Transfer of Samples in Reinforcement Learning
TLDR
A model-based technique that automatically estimates the relevance (importance weight) of each source sample for solving the target task and empirically compares the proposed algorithm to state-of-the-art approaches shows that it achieves better learning performance and is very robust to negative transfer.
Transferring task models in Reinforcement Learning agents
TLDR
This work proposes a novel method for transferring models to Reinforcement Learning agents, implementing both model-free and model-based learning, in order to fully exploit the presence of a source task model.
Transferring Models in Hybrid Reinforcement Learning Agents
TLDR
This work proposes a novel method for transferring models to a hybrid reinforcement learning agent, where the models of the transition and reward functions of a source task, will be transferred to a relevant but different target task.
Transfer in Reinforcement Learning: A Framework and a Survey
  • A. Lazaric
  • Computer Science
    Reinforcement Learning
  • 2012
TLDR
This chapter provides a formalization of the general transfer problem, the main settings which have been investigated so far, and the most important approaches to transfer in reinforcement learning.
Sequential Transfer in Reinforcement Learning with a Generative Model
TLDR
This work designs an algorithm that quickly identifies an accurate solution by seeking the state-action pairs that are most informative for this purpose and derives PAC bounds on its sample complexity which clearly demonstrate the benefits of using this kind of prior knowledge.
REPAINT: Knowledge Transfer in Deep Reinforcement Learning
TLDR
The experimental results on several benchmark tasks show that REPAINT significantly reduces the total training time in generic cases of task similarity, and when the source tasks are dissimilar to, or sub-tasks of, the target tasks, REPAint outperforms other baselines in both training-time reduction and asymptotic performance of return scores.
REPAINT: Knowledge Transfer in Deep Actor-Critic Reinforcement Learning
TLDR
A REPresentation-And-INstance Transfer algorithm (REPAINT) for deep actor-critic reinforcement learning paradigm that significantly reduces the total training time and improves the asymptotic performance compared to training with no prior knowledge and other baselines.
Learning Inter-Task Transferability in the Absence of Target Task Samples
TLDR
This paper proposes a framework for selecting source tasks in the absence of a known model or target task samples and uses meta-data associated with each task to learn the expected benefit of transfer given a source-target task pair.
Transfer Learning for continuous State and Action Spaces
TLDR
This work presents a novel approach to transfer knowledge between tasks in a reinforcement learning (RL) framework with continuous states and actions, where the transition and policy functions are approximated by Gaussian processes.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 15 REFERENCES
Transfer in variable-reward hierarchical reinforcement learning
TLDR
This paper formally defines the transfer learning problem in the context of RL as learning an efficient algorithm to solve any SMDP drawn from a fixed distribution after experiencing a finite number of them, and introduces an online algorithm that compactly stores the optimal value functions for several SMDPs, and uses them to optimally initialize the value function for a new SMDP.
Transferring Instances for Model-Based Reinforcement Learning
TLDR
It is demonstrated that timbrel can significantly improve the sample efficiency and asymptotic performance of a model-based algorithm when learning in a continuous state space.
Tree-Based Batch Mode Reinforcement Learning
TLDR
Within this framework, several classical tree-based supervised learning methods and two newly proposed ensemble algorithms, namely extremely and totally randomized trees, are described and found that the ensemble methods based on regression trees perform well in extracting relevant information about the optimal control policy from sets of four-tuples.
Transfer Learning via Inter-Task Mappings for Temporal Difference Learning
TLDR
This article compares learning on a complex task with three function approximators, a cerebellar model arithmetic computer (CMAC), an artificial neural network (ANN), and a radial basis function (RBF), and empirically demonstrates that directly transferring the action-value function can lead to a dramatic speedup in learning with all three.
Building Portable Options: Skill Transfer in Reinforcement Learning
TLDR
This work introduces the notion of learning options in agentspace, the space generated by a feature set that is present and retains the same semantics across successive problem instances, rather than in problemspace.
Transfer Learning with an Ensemble of Background Tasks
TLDR
The transfer of learning from an ensemble of background tasks is demonstrated, which becomes helpful in cases where a single background task does not transfer well, and whether a useful prior from those multiple task As that gives effective guidance when learning task B is studied.
Reinforcement Learning in Continuous Action Spaces through Sequential Monte Carlo Methods
TLDR
A novel actor-critic approach in which the policy of the actor is estimated through sequential Monte Carlo methods, and results obtained in a control problem consisting of steering a boat across a river are reported.
Task similarity measures for transfer in reinforcement learning task libraries
  • J. L. Carroll, K. Seppi
  • Computer Science, Mathematics
    Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005.
  • 2005
TLDR
Several possible task similarity measures are proposed, based on the transfer time, policy overlap, Q-values, and reward structure respectively, and they are evaluated in three separate experimental situations.
Model-based function approximation in reinforcement learning
TLDR
Preliminary experiments with a novel algorithm, AMBI (Approximate Models Based on Instances), demonstrate that this approach yields faster learning on some standard benchmark problems than many contemporary algorithms.
Model Transfer for Markov Decision Tasks via Parameter Matching
Model transfer refers to the process of transferring information from a model that was previously identified for one task (source) to a new task (target). For decision tasks in unknown Markov
...
1
2
...