Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems

  title={Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems},
  author={Yewen Fan and Nian Si and Kun Zhang},
Calibration is defined as the ratio of the average predicted click rate to the true click rate. The optimization of calibration is essential to many online advertising recommendation systems because it directly affects the downstream bids in ads auctions and the amount of money charged to advertisers. Despite its importance, calibration optimization often suffers from a problem called “maximization bias”. Maximization bias refers to the phenomenon that the maximum of predicted values… 


Calibrating User Response Predictions in Online Advertising
A calibration algorithm to exploit implicit properties of predicted probabilities to reduce negative impacts of the data sparsity problem and an estimation model to leverage post-click information to approximate the real delayed user responses are proposed.
Practical Lessons from Predicting Clicks on Ads at Facebook
This paper introduces a model which combines decision trees with logistic regression, outperforming either of these methods on its own by over 3%, an improvement with significant impact to the overall system performance.
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation
It is demonstrated that a low variance is at least as important, as a non-negligible variance introduces the potential for over-fitting in model selection as well as in training the model, and some common performance evaluation practices are susceptible to a form of selection bias as a result of this form of over- fitting and hence are unreliable.
Simulation of Coherent Risk Measures Based on Generalized Scenarios
In financial risk management, coherent risk measures have been proposed as a way to avoid undesirable properties of measures such as value at risk that discourage diversification and do not account
The Elements of Statistical Learning
Chapter 11 includes more case studies in other areas, ranging from manufacturing to marketing research, and a detailed comparison with other diagnostic tools, such as logistic regression and tree-based methods.
An Upper Confidence Bound Approach to Estimating Coherent Risk Measures
This paper considers the set of probability measures is finite, and study the estimation of a coherent risk measure via an upper confidence bound (UCB) approach, where samples of the portfolio loss are simulated sequentially from one of the probability measures.
Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers
Extensive experiments show that beta calibration is superior to logistic calibration for Naive Bayes and Adaboost and derive the method from first principles and show that fitting it is as easy as fitting a logistic curve.
Verified Uncertainty Calibration
The scaling-binning calibrator is introduced, which first fits a parametric function to reduce variance and then bins the function values to actually ensure calibration, and estimates a model's calibration error more accurately using an estimator from the meteorological community.
Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Average
No unbiased estimator exists and it is non-trivial to select a good estimator without knowledge about the distributions of the random variables, and the bias and variance of the aforementioned estimators are investigated and bound.
Predicting good probabilities with supervised learning
We examine the relationship between the predictions made by different learning algorithms and true posterior probabilities. We show that maximum margin methods such as boosted trees and boosted