• Corpus ID: 6207952

Thompson sampling with the online bootstrap

  title={Thompson sampling with the online bootstrap},
  author={Dean Eckles and Maurits Kaptein},
Thompson sampling provides a solution to bandit problems in which new observations are allocated to arms with the posterior probability that an arm is optimal. While sometimes easy to implement and asymptotically optimal, Thompson sampling can be computationally demanding in large scale bandit problems, and its performance is dependent on the model fit to the observed data. We introduce bootstrap Thompson sampling (BTS), a heuristic method for solving bandit problems which modifies Thompson… 

Figures from this paper

Bootstrap Thompson Sampling and Sequential Decision Problems in the Behavioral Sciences
The utility of bootstrap Thompson sampling (BTS), which replaces the posterior distribution with the bootstrap distribution, is shown and its robustness to model misspecification is illustrated, which is a common concern in behavioral science applications.
Debiasing Samples from Online Learning Using Bootstrap
This paper provides a procedure to debias the samples using bootstrap, which doesn’t require the knowledge of the reward distribution and can be applied to any adaptive policies.
New Insights into Bootstrapping for Bandits
This work shows that the commonly used non-parametric bootstrapping (NPB) procedure can be provably inefficient and establishes a near-linear lower bound on the regret incurred by it under the bandit model with Bernoulli rewards, and proposes a weighted bootstrapped (WB) procedure.
Residual Bootstrap Exploration for Stochastic Linear Bandit
A theoretical framework is contributed to demystify residual bootstrap-based exploration mechanisms in stochastic linear bandit problems and show the significant computational e-ciency of LinReBoot.
Diffusion Approximations for Thompson Sampling
The weak convergence theory covers both the classical multi-armed and linear bandit settings, and can be used to obtain insight about the characteristics of the regret distribution when there is information sharing among arms, as well as the effects of variance estimation, model mis-specification and batched updates in bandit learning.
Bootstrapped Thompson Sampling and Deep Exploration
This technical note presents a new approach to carrying out the kind of exploration achieved by Thompson sampling, but without explicitly maintaining or sampling from posterior distributions. The
Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
A novel imitation-learning-based algorithm is proposed that distills a TS policy into an explicit policy representation by performing posterior inference and optimization offline, which enables fast online decision-making and easy deployment in mobile and server-based environments.
Robust Contextual Bandits via Bootstrapping
An estimator (evaluated from historical rewards) is developed for the contextual bandit UCB based on the multiplier bootstrapping technique and it is proved that the BootLinUCB has a sub-linear regret upper bound and also conducted extensive experiments to validate its superior performance.
A Tutorial on Thompson Sampling
This tutorial covers the algorithm and its application, illustrating concepts through a range of examples, including Bernoulli bandit problems, shortest path problems, product recommendation, assortment, active learning with neural networks, and reinforcement learning in Markov decision processes.
CORe: Capitalizing On Rewards in Bandit Exploration
A gap-free bound is derived on the n -round regret of CORe in a stochastic linear bandit, where d is the number of features and K is thenumber of arms.


Approximate Bayesian-inference With the Weighted Likelihood Bootstrap
We introduce the weighted likelihood bootstrap (WLB) as a way to simulate approximately from a posterior distribution. This method is often easy to implement, requiring only an algorithm for
Bootstrapping Regression Models
Bootstrapping is a general approach to statistical inference based on building a sampling distribution for a statistic by resampling from the data at hand. The term ‘bootstrapping,’ due to Efron
A modern Bayesian look at the multi-armed bandit
A heuristic for managing multi-armed bandits called randomized probability matching is described, which randomly allocates observations to arms according the Bayesian posterior probability that each arm is optimal.
Exploration-exploitation tradeoff using variance estimates in multi-armed bandits
Bootstrapping data arrays of arbitrary order
In this paper we study a bootstrap strategy for estimating the variance of a mean taken over large multifactor crossed random effects data sets. We apply bootstrap reweighting independently to the
An Empirical Evaluation of Thompson Sampling
Empirical results using Thompson sampling on simulated and real data are presented, and it is shown that it is highly competitive and should be part of the standard baselines to compare against.
Thompson Sampling: An Asymptotically Optimal Finite-Time Analysis
The question of the optimality of Thompson Sampling for solving the stochastic multi-armed bandit problem is answered positively for the case of Bernoulli rewards by providing the first finite-time analysis that matches the asymptotic rate given in the Lai and Robbins lower bound for the cumulative regret.
Bandit problems and the exploration/exploitation tradeoff
An analytically simple bandit model is provided that is more directly applicable to optimization theory than the traditional bandit problem and a near-optimal strategy is determined for that model.
The KL-UCB Algorithm for Bounded Stochastic Bandits and Beyond
It is proved that for arbitrary bounded rewards, the KL-UCB algorithm satisfies a uniformly better regret bound than UCB or UCB2; second, in the special case of Bernoulli rewards, it reaches the lower bound of Lai and Robbins.