• Corpus ID: 235435944

Thompson Sampling for Unimodal Bandits

@article{Yang2021ThompsonSF,
  title={Thompson Sampling for Unimodal Bandits},
  author={Long Yang and Zhao Li and Zehong Hu and Shasha Ruan and Shijian Li and Gang Pan and Hongyang Chen},
  journal={ArXiv},
  year={2021},
  volume={abs/2106.08187}
}

• Long Yang, Zhao Li, +4 authors Hongyang Chen
• Published 15 June 2021
• Computer Science
• ArXiv

In this paper, we propose a Thompson Sampling algorithm for unimodal bandits, where the expected reward is unimodal over the partially ordered arms. To exploit the unimodal structure better, at each step, instead of exploration from the entire decision space, our algorithm makes decision according to posterior distribution only in the neighborhood of the arm that has the highest empirical mean estimate. We theoretically prove that, for Bernoulli rewards, the regret of our algorithm reaches the lower bound of unimodal bandits, thus it is asymptotically optimal. For Gaussian rewards, the regret of our algorithm is O(log T ), which is far better than standard Thompson Sampling algorithms. Extensive experiments demonstrate the effectiveness of the proposed algorithm on both synthetic data sets and the real-world applications. Introduction In this paper, we focus on an important bandit: unimodal multi-arm bandit (unimodal MAB) (Yu and Mannor 2011). The unimodal MAB considers a unimodal structure, where there is a path that leads to the unique optimal arm, and along such path, the expected reward is monotonic. As the standard MAB problem, one of the key challenges in the unimodal MAB problem is that the agent must face the trade-off between exploration (gaining information about the potential arm that may yield high payouts in the future) and exploitation (using the current information greedily). The optimistic approaches such as UCB (Lai and Robbins 1985), UCB1 (Auer et al. 2002), KL-UCB (Garivier and Cappé 2011; Cappé et al. 2013), and MOSS (Audibert and Bubeck 2010) are widely used to solve MAB problems. Similarly, to solve the exploration-exploitation dilemma in unimodal bandits, Yu and Mannor (2011) propose the LSE algorithm that bases on Kiefer’s golden search method (Kiefer and Wolfowitz 1952). Later, Combes and Proutiere (2014a) propose OSUB that uses KL-divergence upper confidence bounds to define an index for each arm to search the best arm. However, recently, lots of works, e.g., Russo and Van Roy (2014); Russo and Van Roy (2016); Zhu and Tan (2020), have empirically or theoretically demonstrated that Bayesian methods such as Thompson Sampling is more effective than optimistic approaches. It has been shown that Thompson Copyright © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Sampling usually suffers the same order of regret as optimistic approaches (Kaufmann et al. 2012; Agrawal and Goyal 2013), but it significantly outperforms many optimistic approaches in practice. For example, for the application of display advertising and news article recommendation, Chapelle and Li (2011); Gopalan et al. (2014) show that Thompson Sampling is competitive to or better than the popular UCB. Besides, the experiment of Chapelle and Li (2011) shows that Thompson Sampling is more robust to delayed or batched feedback system, which is out of reach for the optimistic approach. Our Works To improve the existing optimistic unimodal bandit algorithms, we propose TS-UB, which is a novel Thompson Sampling algorithm that keeps not only the posterior distribution but also empirical mean estimates of the expected rewards of each arm. To learn the optimal arm quickly, the proposed TS-UB makes use of the unimodal structure: instead of exploration from the decision space (i.e., the set of all the arms), TS-UB explores the arms from the neighborhood of the current leader (i.e., the arm that has the highest empirical reward). TS-UB keeps the advantage of the simplicity of implementation, and extensive experiments show that TSUB improves the performance of the traditional Thompson Sampling significantly. Concretely, we list our main contributions in the following three aspects. Firstly, TS-UB (Algorithm 2) provides an implementation to solve unimodal bandit with Thompson Sampling. Corollary 1 shows that TS-UB (Algorithm 2) with Beta prior is an asymptotically optimal algorithm. Notably, although both OSUB and UTS are all asymptotically optimal, the regret of TS-UB is never worse than theirs, see Remark 2. Later, we extend Thompson Sampling to unimodal bandits with Gaussian rewards. Theorem 3 shows that the regret of TS-UB (Algorithm 3) reaches O(log T ), which achieves the same order of the optimal regret. It is noteworthy that the standard Thompson Sampling (without unimodal structure) with Gaussian prior achieves a near-optimal regret of O( √ KT logK) (Agrawal and Goyal 2013). Thus, our TSUB improves the result of (Agrawal and Goyal 2013) significantly since we consider the additional unimodal structure. For more discussion, see Remark 3. Finally, we use experiments to support our theory results. ar X iv :2 10 6. 08 18 7v 1 [ cs .L G ] 1 5 Ju n 20 21 Especially, we apply TS-UB to the task of reducing the impressions of the fraud sellers in e-commerce. We model such a task as an unimodal bandit problem, then applying TS-UB to solve it. So far, the manual investigation is the widely used way to such a task in e-commerce (Li et al. 2019), which is inefficient and can only cover a small fraction of potential fraud sellers. The result shows that our method outperforms the manual investigation significantly. Related Works and Comparison Unimodal bandit is a specific instance of multi-arms bandits in metric space (Kleinberg 2004; Kleinberg et al. 2008), and it has received relatively little attention. Cope (2009) and Yu and Mannor (2011) have addressed the unimodal bandit with continuous arms. Cope (2009) proposes a method via Kiefer-Wolfowitz stochastic approximation (Kiefer and Wolfowitz 1952). Under the assumption that the reward function is strongly convex, Cope (2009) proves its algorithm achieves a regret of O( √ T ). Later, with the knowledge of the smoothness of the reward function, Yu and Mannor (2011) propose the line search elimination (LSE) algorithm that achieves a regret of O( √ T log T ). Then, Combes and Proutiere (2014b) propose the stochastic pentachotomy (SP) algorithm that achieves the same regret as Yu and Mannor (2011), but no further assumption is made concerning the smoothness or the structure of the reward function. For the unimodal bandit with discrete arms, Combes and Proutiere (2014a) shows the lower bound of the unimodal bandit problem, which will be presented in the next section of our paper, see Eq.(1). Besides, Combes and Proutiere (2014a) propose OSUB algorithm that matches this lower bound for Bernoulli reward. The OSUB is based on KL-UCB (Garivier and Cappé 2011; Cappé et al. 2013), and it uses KL-divergence upper confidence bounds to define an index for each arm. Later, Paladino et al. (2017) inherits the selecting leader rule of OSUB, and design the UTS that is based on Thompson Sampling to solve the unimodal bandit problem. Recently, Trinh et al. (2020) introduce Thompson Sampling to solve rank-one unimodal bandits. Comparison One of the major differences between our TS-UB and UTS / OSUB is that our TS-UB removes the step of playing the current leader by a fixed frequency. Setting a frequency parameter manually is a cumbersome task in practice. Although both OSUB and UTS are all asymptotically optimal, the regret of TS-UB is never worse than theirs, please see Remark 2. Thus, our TS-UB is simpler for implementation but more effective than UTS and OSUB theoretically. Preliminary We consider the stochastic multi-bandit problem with K arms. At each round t ∈ N, the agent selects an arm a ∈ [K] =: {1, 2, · · · ,K}. Then the agent receives a random real-valued reward Xa,t according to an unknown (fixed) distribution νa with the expectation μa = E[Xa,t]. We assume that there is an unique optimal arm a satisfies μ =: μa? = maxa∈[K] μa. We use ∆a =: μ − μa to denote the mean gap between the optimal arm a and a. Unimodal Structure In this paper, we mainly focus on the unimodal bandit problem where the expected reward function is unimodal over a partial order set of arms (Yu and Mannor 2011; Combes and Proutiere 2014a). Let G = (V, E) be an undirected graph, where V = [K] is the set of vertices correspond to arms, and an edge (i, j) ∈ E characterizes a partial order between the expected rewards of the arm i and j. Let ν = (ν1, · · · , νK) denote the joint reward distribution, the partial order of G is unknown since ν is a prior unknown. We make a precise expression of the unimodal structure in the Assumption 1. Assumption 1 ((Combes and Proutiere 2014a)). Let G = (V, E) denote an undirected graph. The expected reward function is unimodal along every path P = (a1, · · · , am) contains a, such that μa1 < μa2 < · · · < μa? , μa? > · · · > μam−1 > μam , where (ai, ai+1) ∈ E , 1 ≤ i ≤ m− 1. The above unimodality is quite general, and it includes the special case: unimodal function (where G is just a line). We use N (i) to denote the neighborhood of arm i, specifically, N (i) = {j|(i, j) ∈ E}. Without loss of generality, we assume E does not contain self-edges (i, i) since it does not contribute to increasing paths. The extended neighborhood of arm i is defined as N(i) = N (i) ∪ {i}. Regret Let A = {at}t=1 be generated by an algorithm π, where at ∈ [K]. The regret of π is defined as R(T ) = Tμ − E[ T ∑ t=1 Xat,t], where the expectation E[·] is taken with respect to the measure on outcomes induced by the interaction of the policy π and the joint reward distribution ν. Without loss of generality, we consider the uniformly good policy (Lai and Robbins 1985), i.e., R(T ) = o(T ), for some a > 0. Uniformly good algorithms always exist; e.g, UCB is uniformly good. Theorem 1. (Combes and Proutiere 2014a) Let π be a uniformly good policy. Under Assumption 1, for a given unimodal bandit G = (V, E), the following holds, lim T→∞ inf R(T ) log(T ) ≥ ∑ a∈N (a?) μ − μa KL(μa|μ) , (1) where KL(μa|μ) = μa log μa μ? + (1− μa) log 1−μa 1−μ? is KLdivergence value between Bernoulli distributions of respective means μa and μ. Theorem 1 illustrates the lower bound regret of unimodal ban