Estimating Position Bias without Intrusive Interventions

@article{Agarwal2019EstimatingPB,
  title={Estimating Position Bias without Intrusive Interventions},
  author={Aman Agarwal and Ivan Zaitsev and Xuanhui Wang and Cheng Li and Marc Najork and Thorsten Joachims},
  journal={Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining},
  year={2019}
}
Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal. [...] Key Method First, we show how to harvest a specific type of intervention data from historic feedback logs of multiple different ranking functions, and show that this data is sufficient for consistent propensity estimation in the position-based model. Second, we propose a new extremum estimator that makes effective use of this data. In an empirical evaluation, we find…Expand
Counterfactual Learning to Rank using Heterogeneous Treatment Effect Estimation
TLDR
This work employs heterogeneous treatment effect estimation techniques to estimate position bias when intervention click data is limited and uses such estimations to debias the observed click distribution and re-draw a new de-biased data set, which can be used for any LTR algorithms. Expand
Intervention Harvesting for Context-Dependent Examination-Bias Estimation
TLDR
A Contextual Position-Based Model (CPBM) where the examination bias may also depend on a context vector describing the query and the user is proposed, and an effective estimator for the CPBM based on intervention harvesting is proposed. Expand
Ranker-agnostic Contextual Position Bias Estimation
TLDR
This paper introduces a method for modeling the probability of an item being seen in different contexts, e.g., for different users, with a single estimator, and indicates that the method introduced outperforms other existing position bias estimators in terms of relative error when the examination probability varies across queries. Expand
Position Bias Estimation for Unbiased Learning-to-Rank in eCommerce Search
TLDR
A novel method to directly estimate propensities which does not use any intervention in live search or rely on modeling relevance, and is applied to any search engine for which the rank of the same document may naturally change over time for the same query. Expand
Direct Estimation of Position Bias for Unbiased Learning-to-Rank without Intervention
TLDR
A novel method to directly estimate propensities which does not use any intervention in live search or rely on modeling relevance, and is applied to any search engine for which the rank of the same document may naturally change over time for the same query. Expand
Adapting Interactional Observation Embedding for Counterfactual Learning to Rank
TLDR
This work uses the embedding method to develop an Interactional Observation-Based Model (IOBM) and argues that while there exist complex observed and unobserved confounders for observation/click interactions, it is sufficient to use theembedding as a proxy confounder to uncover the relevant information for the prediction of the observation propensity. Expand
Propensity-Independent Bias Recovery in Offline Learning-to-Rank Systems
TLDR
A new counterfactual method is proposed that uses a two-stage correction approach and jointly addresses selection and position bias in learning-to-rank systems without relying on propensity scores, and is better than state-of-the-art propensity-independent methods and either better than or comparable to methods that make the strong assumption for which the propensity model is known. Expand
Unifying Online and Counterfactual Learning to Rank
Optimizing ranking systems based on user interactions is a wellstudied problem. State-of-the-art methods for optimizing ranking systems based on user interactions are divided into online approaches –Expand
Unifying Online and Counterfactual Learning to Rank: A Novel Counterfactual Estimator that Effectively Utilizes Online Interventions
TLDR
The introduction of the intervention-aware estimator is proposed, which aims to bridge the online/counterfactual LTR division as it is shown to be highly effective in both online and counterfactual scenarios. Expand
Unbiased Learning to Rank via Propensity Ratio Scoring
TLDR
This work proves that in a LTR algorithm that is based on pairwise comparisons, only pairs with different labels should contribute to the loss function and derives a new weighting scheme called Propensity Ratio Scoring (PRS), which ensures a more effective use of click data in various situations, which leads to its superior performance in an extensive set of LTR benchmarks. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 41 REFERENCES
Effective Evaluation Using Logged Bandit Feedback from Multiple Loggers
TLDR
This paper finds that the standard Inverse Propensity Score (IPS) estimator suffers especially when logging and target policies diverge -- to a point where throwing away data improves the variance of the estimator. Expand
Position Bias Estimation for Unbiased Learning to Rank in Personal Search
TLDR
This paper proposes a regression-based Expectation-Maximization (EM) algorithm that is based on a position bias click model and that can handle highly sparse clicks in personal search and compares the pointwise and pairwise learning-to-rank models. Expand
Unbiased Learning-to-Rank with Biased Feedback
TLDR
A counterfactual inference framework is presented that provides the theoretical basis for unbiased LTR via Empirical Risk Minimization despite biased data, and a Propensity-Weighted Ranking SVM is derived for discriminative learning from implicit feedback, where click models take the role of the propensity estimator. Expand
The Self-Normalized Estimator for Counterfactual Learning
This paper identifies a severe problem of the counterfactual risk estimator typically used in batch learning from logged bandit feedback (BLBF), and proposes the use of an alternative estimator thatExpand
Unbiased Learning to Rank with Unbiased Propensity Estimation
TLDR
DLA is an automatic unbiased learning-to-rank framework as it directly learns unbiased ranking models from biased click data without any preprocessing and it can adapt to the change of bias distributions and is applicable to online learning. Expand
Counterfactual Learning-to-Rank for Additive Metrics and Deep Models
TLDR
This work generalizes the counterfactual learning-to-rank approach to a broad class of additive rank metrics -- like Discounted Cumulative Gain (DCG) and Precision@k -- as well as non-linear deep network models, and develops two new learning methods that both directly optimize an unbiased estimate of DCG despite the bias in the implicit feedback data. Expand
An experimental comparison of click position-bias models
TLDR
A cascade model, where users view results from top to bottom and leave as soon as they see a worthwhile document, is the best explanation for position bias in early ranks. Expand
Batch learning from logged bandit feedback through counterfactual risk minimization
TLDR
The empirical results show that the CRM objective implemented in POEM provides improved robustness and generalization performance compared to the state-of-the-art, and a decomposition of the POEM objective that enables efficient stochastic gradient optimization is presented. Expand
Offline Comparative Evaluation with Incremental, Minimally-Invasive Online Feedback
TLDR
This work investigates the use of logged user interaction data---queries and clicks---for offline evaluation of new search systems in the context of counterfactual analysis and presents a methodology for incrementally logging interactions on previously-unseen documents for use in computation of an unbiased estimator of a new ranker's effectiveness. Expand
Beyond position bias: examining result attractiveness as a source of presentation bias in clickthrough data
TLDR
This study distinguishes itself from prior work by aiming to detect systematic biases in click behavior due to attractive summaries inflating perceived relevance, and shows substantial evidence of presentation bias in clicks towards results with more attractive titles. Expand
...
1
2
3
4
5
...