• Corpus ID: 250451656

A Dataset Perspective on Offline Reinforcement Learning

@inproceedings{Schweighofer2021ADP,
  title={A Dataset Perspective on Offline Reinforcement Learning},
  author={Kajetan Schweighofer and Andreas Radler and Marius-Constantin Dinu and Markus Hofmarcher and Vihang Patil and Angela Bitto-Nemling and Hamid Eghbal-zadeh and Sepp Hochreiter},
  year={2021}
}

• Kajetan Schweighofer, Andreas Radler, +5 authors S. Hochreiter
• Published 8 November 2021
• Computer Science

The application of Reinforcement Learning (RL) in real world environments can be expensive or risky due to sub-optimal policies during training. In Offline RL, this problem is avoided since interactions with an environment are prohibited. Policies are learned from a given dataset, which solely determines their performance. Despite this fact, how dataset characteristics influence Offline RL algorithms is still hardly investigated. The dataset characteristics are determined by the behavioral policy that samples this dataset. Therefore, we define characteristics of behavioral policies as exploratory for yielding high expected information in their interaction with the Markov Decision Process (MDP) and as exploitative for having high expected return. We implement two corresponding empirical measures for the datasets sampled by the behavioral policy in deterministic MDPs. The first empirical measure SACo is defined by the normalized unique state-action pairs and captures exploration. The second empirical measure TQ is defined by the normalized average trajectory return and captures exploitation. Empirical evaluations show the effectiveness of TQ and SACo. In large-scale experiments using our proposed measures, we show that the unconstrained off-policy Deep Q-Network family requires datasets with high SACo to find a good policy. Furthermore, experiments show that policy constraint algorithms perform well on datasets with high TQ and SACo. Finally, the experiments show, that purely dataset-constrained Behavioral Cloning performs competitively to the best Offline RL algorithms for datasets with high TQ. study, we derived two theoretical measures, the transition-entropy corresponding to explorativeness of a policy and the expected trajectory return, corresponding to the exploitativeness of a policy. Furthermore, we analyzed stability traits of these measures under MDP homomorphisms. Moreover, we implemented two empirical measures, TQ and SACo, which correspond to the expected trajectory return and the transition-entropy in deterministic MDPs. We generated 150 datasets using six environments and five dataset sampling strategies over five seeds. On these datasets, the performance of nine model-free algorithms was evaluated over independent runs, resulting in 6750 trained offline policies. The performance of the offline algorithms shows clear correlations with the exploratory and exploitative characteristics of the dataset, measured by TQ and SACo. We found, that popular algorithms in Offline RL are strongly influenced by the characteristics of the dataset and the average performance across different datasets might not be enough for a fair comparison. Our study thus provides a blueprint to characterize Offline RL datasets and understanding their effect on algorithms.