A Robust UCB scheme for active learning in regression from strategic crowds

  title={A Robust UCB scheme for active learning in regression from strategic crowds},
  author={Divya Padmanabhan and Satyanath Bhat and Dinesh Garg and Shirish K. Shevade and Y. Narahari},
  journal={2016 International Joint Conference on Neural Networks (IJCNN)},
We study the problem of training an accurate linear regression model by procuring labels from multiple noisy crowd annotators, under a budget constraint. We propose a Bayesian model for linear regression in crowdsourcing and use variational inference for parameter estimation. To minimize the number of labels crowdsourced from the annotators, we adopt an active learning approach. In this specific context, we prove the equivalence of well-studied criteria of active learning like entropy… Expand
Corruption-tolerant bandit learning
This work proposes algorithms that use recent advances in robust statistical estimation to perform arm selection in polynomial time and vastly outperform several existing UCB and EXP-style algorithms for stochastic and adversarial multi-armed and linear-contextual bandit problems in wide variety of experimental settings. Expand
Dominant strategy truthful, deterministic multi-armed bandit mechanisms with logarithmic regret for sponsored search auctions
A dominant strategy incentive compatible (DSIC) and individually rational (IR), deterministic MAB mechanism, based on ideas from the Upper Confidence Bound (UCB) family of MAB algorithms, achieves a Δ-regret of $O(\log T)$ for the case of sponsored search auctions. Expand
Theoretical Models for Learning from Multiple, Heterogenous and Strategic Agents
This work broadly study three problems in the context of learning from multiple agents, (1) Multi-label classification (2) Active Linear Regression (3) Sponsored Search Auctions. Expand


Bayesian Bias Mitigation for Crowdsourcing
This work presents Bayesian Bias Mitigation for Crowdsourcing (BBMC), a Bayesian model to unify all three steps of data curation and learning and proposes a general approximation strategy for Markov chains to efficiently quantify the effect of a perturbation on the stationary distribution. Expand
Sequential crowdsourced labeling as an epsilon-greedy exploration in a Markov Decision Process
Experimental results confirm that the proposed sequential labeling procedure can achieve similar accuracy at roughly half the labeling cost and at any stage in the labeling process the algorithm achieves a higher accuracy compared to randomly asking for the next label. Expand
Gaussian Process Classification and Active Learning with Multiple Annotators
This paper generalizes GP classification in order to account for multiple annotators with different levels expertise, and empirically shows that the model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Expand
Learning From Crowds
A probabilistic approach for supervised learning when the authors have multiple annotators providing (possibly noisy) labels but no absolute gold standard, and experimental results indicate that the proposed method is superior to the commonly used majority voting baseline. Expand
Learning to Predict from Crowdsourced Data
A novel mixture model is employed for worker annotations, which learns a prediction model directly from samples to labels for efficient out-of-sample testing. Expand
Active Learning with Distributional Estimates
This paper derives a novel AL scheme that balances the current decision boundary and exploration of poorly sampled regions in a natural way, and develops a corresponding AL scheme, where the uncertainty in ^p(y|x) is modeled by a second-order distribution. Expand
Maximizing Expected Model Change for Active Learning in Regression
  • Wenbin Cai, Ya Zhang, Jun Zhou
  • Computer Science
  • 2013 IEEE 13th International Conference on Data Mining
  • 2013
A new active learning framework for regression called Expected Model Change Maximization (EMCM) is proposed, which aims to choose the examples that lead to the largest change to the current model. Expand
Learning from Multiple Annotators with Gaussian Processes
A Gaussian process (GP) approach to regression with multiple labels but no absolute gold standard provides a principled non-parametric framework that can automatically estimate the reliability of individual annotators from data without the need of prior knowledge. Expand
Adaptive Crowdsourcing Algorithms for the Bandit Survey Problem
This work proposes a simple model for adaptive quality control in crowdsourced multiple-choice tasks which it calls the bandit survey problem and presents several algorithms for this problem, based in the experience conducting relevance evaluation for a large commercial search engine. Expand
Truthful Interval Cover Mechanisms for Crowdsourcing Applications
It is shown that the task allocation problem is polynomial time solvable in the homogeneous case while it is NP-hard in the heterogeneous case, and a novel approximation algorithm is proposed that is monotone, leading to a truthful interval cover mechanism via appropriate payments. Expand