Generalized Bias-Variance Evaluation of TREC Participated Systems

@inproceedings{Zhang2014GeneralizedBE,
  title={Generalized Bias-Variance Evaluation of TREC Participated Systems},
  author={Peng Zhang and Linxue Hao and Dawei Song and Jun Wang and Yuexian Hou and Bin Hu},
  booktitle={CIKM},
  year={2014}
}
Recent research has shown that the improvement of mean retrieval effectiveness (e.g., MAP) may sacrifice the retrieval stability across queries, implying a tradeoff between effectiveness and stability. The evaluation of both effectiveness and stability are often based on a baseline model, which could be weak or biased. In addition, the effectiveness-stability tradeoff has not been systematically or quantitatively evaluated over TREC participated systems. The above two problems, to some extent… CONTINUE READING