• Corpus ID: 232380390

Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow

@article{Mcleod2021BellmanAT,
  title={Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow},
  author={John Mcleod and Hrvoje Stoji{\'c} and Vincent Adam and Dongho Kim and Jordi Grau-Moya and Peter Vrancx and Felix Leibfried},
  journal={ArXiv},
  year={2021},
  volume={abs/2103.14407}
}

• John Mcleod, Hrvoje Stojić, +4 authors Felix Leibfried
• Published 26 March 2021
• Computer Science
• ArXiv

In the past decade, model-free reinforcement learning (RL) has provided solutions to challenging domains such as robotics. Model-based RL (where agents learn a model of the environment in order to explicitly plan ahead) shows the prospect of being more sampleefficient than model-free methods in terms of agent-environment interactions, because the model enables to extrapolate to unseen situations. In the more recent past, model-based methods have shown superior results compared to model-free methods in some challenging domains with non-linear state transitions. At the same time, it has become apparent that RL is not market-ready yet and that many real-world applications are going to require model-based approaches, because model-free methods are too sample-inefficient and show poor performance in early stages of training. The latter is particularly important in industry, e.g. in production systems that directly impact a company’s revenue. This demonstrates the necessity for a toolbox to push the boundaries for model-based RL. While there is a plethora of toolboxes for model-free RL, model-based RL has received little attention in terms of toolbox development. Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox using state-of-the-art software engineering practices. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms. We also provide an experiment harness to compare both model-free and model-based agents in a systematic fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This paves the way for new research directions, e.g. investigating uncertainty-aware environment models that are not necessarily neural-network-based, or developing algorithms to solve industrially-motivated benchmarks that share characteristics with real-world problems. ∗. Work done while at Secondmind. 1 ar X iv :2 10 3. 14 40 7v 2 [ cs .L G ] 1 3 A pr 2 02 1 J. McLeod, H. Stojić, V. Adam, D. Kim, J. Grau-Moya, P. Vrancx and F. Leibfried 1. Existing Toolboxes for Reinforcement Learning In recent years, plenty of toolboxes with state-of-the-art model-free RL algorithms have been developed. We enumerate some of them in the following but do not claim this to be an exhaustive list. Amongst the earliest were “RL-Glue” (Tanner and White, 2009), “rllab” (Duan et al., 2016) and “keras-rl” (Plappert, 2016), followed by “Reinforcement Learning Coach” (Caspi et al., 2017), “OpenAI Baselines” (Dhariwal et al., 2017) and “Tensorforce” (Kuhnle et al., 2017), that were later succeeded by “Stable Baselines”(Hill et al., 2018) and “RLlib”(Liang et al., 2018). At the time of writing, “TF-Agents”(Guadarrama et al., 2018) in TensorFlow (Abadi et al., 2016) and “Horizon” (Gauci et al., 2019) in PyTorch (Paszke et al., 2019) appeared to be the most stable and well-maintained toolboxes. On the model-based RL side, there are two recent projects that stick out. Wang et al. (2019) provide a manuscript plus open-sourced code to evaluate various contemporary model-free and model-based RL algorithms on 18 benchmarks, and Dong et al. (2020) provide a software framework called “Baconian” (with some algorithms being completed and some still being developed). While the former focuses more on the research aspect and the second more on the software aspect, what both have in common is that algorithms are implemented in a stand-alone fashion in separate classes without modular & hierarchical object-oriented abstractions and without the flexibility to design new algorithms. 2. Motivation for a Model-Based Reinforcement Learning Toolbox Recently, model-based RL methods have been developed that outperform model-free methods in challenging high-dimensional domains (Wang et al., 2019). At the same time, the community became aware of issues that arise when applying RL in practice, and there is an implicit understanding that many real-world problems are going to require model-based approaches (Dulac-Arnold et al., 2019) for sample-efficiency reasons and because simulators of industrial systems are not available. A second observation is that the literature presents algorithms in a standalone fashion because they are usually developed by different groups, and frameworks are designed in such a way that individual algorithms are implemented in separate classes without inheriting from well-designed object-oriented superclasses that exploit commonalities between algorithms (Wang et al., 2019; Dong et al., 2020). It is often difficult to tell which precise combination of algorithmic components yields superior results: is it the environment model, the decision-making component, or the choice of optimization objectives? The fact that different papers report results under different environments, reward functions or experiment protocols exacerbates the situation (Henderson et al., 2018). The aim of Bellman was to address those problems via identifying commonalities between contemporary model-based RL algorithms and designing a modular software framework where algorithms arise naturally in subclasses and from the composition of well-chosen object-oriented abstractions. This enables the comparison of different algorithms, but importantly, in a controlled setup where the model-based component across different algorithms is the same. Alternatively, different models can be tested simply by changing the environment model object (e.g. by replacing a neural net with a Gaussian process). Furthermore, Bellman provides an experiment harness to train and evaluate model-based and -free algorithms systematically and user-conveniently under the same experiment protocol.