• Corpus ID: 12490849

A Laboratory-Based Method for the Evaluation of Personalised Search

@inproceedings{Sanvitto2016ALM,
  title={A Laboratory-Based Method for the Evaluation of Personalised Search},
  author={Camilla Sanvitto and Debasis Ganguly and G. Jones and Gabriella Pasi},
  booktitle={EVIA@NTCIR},
  year={2016}
}
Comparative evaluation of Information Retrieval Systems (IRSs) using publically available test collections has become an established practice in Information Retrieval (IR). By means of the popular Cranfield evaluation paradigm IR test collections enable researchers to compare new methods to existing approaches. An important area of IR research where this strategy has not been applied to date is Personalised Information Retrieval (PIR), which has generally relied on user-based… 

Figures from this paper

A Proposed Method for Laboratory-Based Evaluation of Personalised Information Retrieval
TLDR
An experimental framework for the creation of test collections to facilitate repeatable laboratory-based evaluation of PIR, together with a prototype evaluation tool to analyze results collected using these collections are introduced.
Evaluation in Contextual Information Retrieval
TLDR
A critical summary and comparison of existing contextual information retrieval evaluation methodologies and metrics according to a simple stratification model is provided and the impact of context dynamicity and data privacy on the evaluation design is pointed out.
Evaluation of Personalised Information Retrieval at CLEF 2017 (PIR-CLEF): Towards a Reproducible Evaluation Framework for PIR
TLDR
The PIR-CLEF 2017 workshop is intended to review the design and construction of this Pilot collection and to consider the topic of reproducible evaluation of PIR more generally with the aim of launching a more formal PIR Lab at CLEF 2018.
Report on the CHIIR 2019 Second Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2019)
The Second Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2019) was held in conjunction with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR
Adapting Information Retrieval to User Signals via Stochastic Models
TLDR
A new family of evaluation measures, called Markov Precision (MP) is presented, based on the proposed model and able to explicitly link lab-style and on-line evaluation metrics and will define a collection for evaluation and comparison of Personalized Information Retrieval (PIR) systems.
WEPIR 2018: Workshop on Evaluation of Personalisation in Information Retrieval
TLDR
The planned outcome of the workshop is the proposal of methodologies to support evaluation of personalisation in information retrieval from both the perspectives of the user experience in interactive search settings, and of user models for personalised information retrieval and their algorithmic incorporation in the search process.
Second Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2019)
TLDR
The planned outcome of the workshop is to progress the work from WEPIR 2018 towards the development of concrete proposals for novel and innovative methodologies to support evaluation of personalisation in information retrieval from both the perspectives of the user experience in interactive search settings and of user models for personalised information retrieval and their algorithmic incorporation in the search process.
Report on the CHIIR 2021 third workshop on evaluation of personalisation in information retrieval (WEPIR 2021)
The Third Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2021) was held in conjunction with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR
Evaluation of Personalised Information Retrieval at CLEF 2018 (PIR-CLEF)
TLDR
The PIR-CLEF 2018 Task is the first PIR evaluation benchmark based on the Cranfield paradigm, with the potential benefits of producing evaluation results that are easily reproducible, and the workshop at the conference will form the basis of further discussion of strategies for the evaluation of PIR.
Overview of the CLEF 2017 Personalised Information Retrieval Pilot Lab (PIR-CLEF 2017)
TLDR
The intention of the collection is to allow research groups working on PIR to both experience with and provide feedback about the proposed PIR evaluation methodology with the aim of launching a more formal PIR Lab at CLEF 2018.
...
...

References

SHOWING 1-10 OF 12 REFERENCES
Evaluation of contextual information retrieval effectiveness: overview of issues and research
TLDR
A comprehensive survey of contextual information retrieval evaluation methodologies is presented and insights into how and why they are appropriate to measure the retrieval effectiveness are provided.
A taxonomy of web search
TLDR
This taxonomy of web searches is explored and how global search engines evolved to deal with web-specific needs is discussed.
Issues in Personalizing Information Retrieval
  • G. Pasi
  • Computer Science
    IEEE Intell. Informatics Bull.
  • 2010
TLDR
The main issues related to the two basic problems beyond these approaches: context representation and definition of processes which exploit the context knowledge to improve the quality of the search outcome are outlined.
HARD Track Overview in TREC 2003: High Accuracy Retrieval from Documents
Abstract : The High Accuracy Retrieval from Documents (HARD) track explores methods for improving the accuracy of document retrieval systems. It does so by considering three questions. Can additional
Overview of the Personalized and Collaborative Information Retrieval (PIR) Track at FIRE-2011
The personalized and collaborative information retrieval (PIR) track at FIRE 2011 was organized with the aim of extending standard information retrieval (IR) ad-hoc test collection design to
Overview of the TREC 2014 Session Track
TLDR
The TREC Session track ran for the fourth time in 2014, and the data includes far more sessions than previous years, with 1,257 unique sessions as compared to around 100 for each of the previous three years.
Searching, browsing, and clicking in a search session: changes in user behavior by task and over time
TLDR
This study characterize and compare user behavior in relatively long search sessions for search tasks of four different types, noting that users shift their interests to focus less on the top results but more on results ranked at lower positions in browsing and that results eventually become less and less attractive for the users.
Overview of the Fourth Text REtrieval Conference (TREC-4)
TLDR
Un resume of cette conference (TREC-4), une description des collections tests utilisees pour les recherches mises en place lors ofertains, and il en souligne les resultats.
Interpolated estimation of Markov source parameters from sparse data
The effect of sampling strategy on inferred measures
TLDR
Examination of the effect on collection quality of different sampling strategies within the inferred measures framework finds strategies that include a top-10 pool create better collections than those that do not, as well as allow Precision(10) scores to be directly computed.
...
...