Relevance Estimation with Multiple Information Sources on Search Engine Result Pages

  title={Relevance Estimation with Multiple Information Sources on Search Engine Result Pages},
  author={Junqi Zhang and Yiqun Liu and Shaoping Ma and Qi Tian},
  journal={Proceedings of the 27th ACM International Conference on Information and Knowledge Management},
  • Junqi Zhang, Yiqun Liu, Qi Tian
  • Published 17 October 2018
  • Computer Science
  • Proceedings of the 27th ACM International Conference on Information and Knowledge Management
Relevance estimation is among the most important tasks in the ranking of search results because most search engines follow the Probability Ranking Principle. Current relevance estimation methodologies mainly concentrate on text matching between the query and Web documents, link analysis and user behavior models. However, users judge the relevance of search results directly from Search Engine Result Pages (SERPs), which provide valuable signals for reranking. Morden search engines aggregate… 

Figures and Tables from this paper

Search Result Reranking with Visual and Structure Information Sources
Experimental results show that the proposed two models achieve better performance than state-of-the-art ranking solutions as well as the original rankings of commercial search engines.
WANDS: Dataset for Product Search Relevance Assessment
This work proposes a systematic and e-ective way to build a discriminative, reusable, and fair human-labeled dataset, Wayfair Annotation DataSet (WANDS), for e-commerce scenarios and introduces an important cross-referencing step to the annotation process which increases dataset completeness.
Context-Aware Ranking by Constructing a Virtual Environment for Reinforcement Learning
A novel framework which aims to improve context-aware listwise ranking performance by optimizing online evaluation metrics and construct a virtual environment with millions of historical click logs to simulate the behavior of real users is proposed.
User Behavior Simulation for Search Result Re-ranking
Two different simulation environments for the offline training of the RL ranking agent are proposed: the Context-aware Click Simulator (CCS) and the Fine-grained User Behavior Simulator with GAN (UserGAN) and a User Behavior Simulation for Reinforcement Learning (UBS4RL) re-ranking framework is designed.
A Context-Aware Click Model for Web Search
A novel neural- based Context-Aware Click Model (CACM) for Web search that outperforms existing click models in both relevance estimation and click prediction tasks.
Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning
This paper decomposes the task of aggregating search results from heterogeneous sources in an E-commerce environment into two subtasks and builds a model with hierarchical reinforcement learning that obtains remarkable improvements in search performance metrics, and achieves a higher user satisfaction.
ViTOR: Learning to Rank Webpages Based on Visual Features
The Visual learning TO Rank (ViTOR) model is introduced that integrates state-of-the-art visual features extraction methods, and it significantly improves the performance of LTR with visual features.
Why Don't You Click: Neural Correlates of Non-Click Behaviors in Web Search
A lab-based user study is designed and differences in brain signals while examining non-click search results with different usefulness levels are revealed and it is shown that it is feasible to utilize brain signals to improve usefulness estimation performance and enhancing human-computer interactions in “Zero-click” search scenarios.
ULTRE framework: a framework for Unbiased Learning to Rank Evaluation based on simulation of user behavior
The proposed ULTRE framework utilizes multiple click models in generating simulated click logs and supports the evaluation of both the offline, counterfactual and the online, bandit-based ULTR models.


Optimizing search engines using clickthrough data
The goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking.
Learning Visual Features from Snapshots for Web Search
This work proposes a novel visual perception model inspired by human's visual search behaviors on page viewing to extract the visual features of Web pages and shows that such visual features can be efficiently acquired in the online setting with an extended inverted indexing scheme.
A user browsing model to predict search engine click data from past observations.
It is confirmed that a user almost always see the document directly after a clicked document, and why documents situated just after a very relevant document are clicked more often is explained.
Ranking Relevance in Yahoo Search
This paper introduces three key techniques for base relevance -- ranking functions, semantic matching features and query rewriting, and describes solutions for recency sensitive relevance and location sensitive relevance.
User Preference Prediction in Mobile Search
A machine learning model to predict user preference automatically in mobile search environment is proposed and it is found that the relevance features can Predict user preference very well, so the agreement of evaluation metrics with side-by-side user preferences on the authors' dataset is compared.
Cumulated gain-based evaluation of IR techniques
This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position, and test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences.
Incorporating vertical results into search click models
Experimental results show that the new Vertical-aware Click Model (VCM) is better at interpreting user click behavior on federated searches in terms of both log-likelihood and perplexity than existing models.
Efficient multiple-click models in web search
This paper presents two multiple-click models: the independent click model which is reformulated from previous work, and the dependent click model (DCM) which takes into consideration dependencies between multiple clicks.
The Probabilistic Relevance Framework: BM25 and Beyond
This work presents the PRF from a conceptual point of view, describing the probabilistic modelling assumptions behind the framework and the different ranking algorithms that result from its application: the binary independence model, relevance feedback models, BM25 and BM25F.
A dynamic bayesian network click model for web search ranking
A Dynamic Bayesian Network is proposed which aims at providing us with unbiased estimation of the relevance from the click logs and shows that the proposed click model outperforms other existing click models in predicting both click-through rate and relevance.