User-centered evaluation of adaptive and adaptable systems: a literature review

@article{Velsen2008UsercenteredEO,
  title={User-centered evaluation of adaptive and adaptable systems: a literature review},
  author={Lex van Velsen and Thea van der Geest and Rob Klaassen and Micha{\"e}l F. Steehouder},
  journal={The Knowledge Engineering Review},
  year={2008},
  volume={23},
  pages={261 - 281}
}
Abstract This literature review focuses on user-centered evaluation (UCE) studies of adaptive and adaptable systems. Usability, perceived usefulness and appropriateness of adaptation are the three most commonly assessed variables. Questionnaires appeared to be the most popular method, followed by interviews and data log analysis. The quality of most questionnaires was questionable, and the reporting of interviews and think-aloud protocols was found to be shallow. Furthermore, data logs need… 
An Evaluation Framework for End-User Experience in Adaptive Systems
TLDR
A web-based framework to support the evaluation of end-user experience in adaptive and personalised systems is introduced based upon advice from domain experts and a review of evaluation approaches, methodologies and techniques adopted by existing adaptive systems.
Testing the usability of a personalized system : comparing the use of interviews , questionnaires and thinking-aloud
TLDR
Conurrent thinking-aloud turned out to be the best method to elicit comments on the perceived relevance of search results and a personalized search engine to compare the use of interviews, questionnaires and concurrent thinkingaloud to this avail.
Testing the usability of a personalized system: comparing the use of interviews, questionnaires and thinking-aloud
TLDR
Conurrent thinking-aloud turned out to be the best method to elicit comments on the perceived relevance of search results and a personalized search engine was evaluated to compare the use of interviews, questionnaires and concurrent thinking- aloud.
Towards comparative evaluations of user-adaptive software systems
TLDR
Taking inspiration from similar research communities and building upon current evaluation approaches, a question arises as to whether it is possible to create such a community and develop a common evaluation mechanism of useradaptive systems.
Identifying Usability Issues for Personalization During Formative Evaluations: A Comparison of Three Methods
TLDR
Questionnaires and interviews appear to be more useful for assessing specific usability issues for personalization, whereas thinking-aloud generates more comments on the usefulness of the system in the intended context of use and identifies the most critical and serious problems.
Workshop on User-Centred Design and Evaluation of Adaptive Systems
TLDR
This talk will first look at some special considerations that arise with user-adaptive systems such as the possibility of using relevant human experts either as judges of the appropriateness of the system’s adaptation or as wizards within the Wizard-of-Oz paradigm.
Context- and Data-driven Satisfaction Analysis of User Interface Adaptations Based on Instant User Feedback
TLDR
An on-the-fly usability testing solution that combines continuous context monitoring together with collection of instant user feedback to assess end-user satisfaction of UI adaptation features and results show that most of the triggeredUI adaptation features were positively rated.
Research through Design: Qualitative Analysis to Evaluate the Usability
TLDR
An assessment of the usability of the qualitative data analysis software webQDA® (version 2.0) is presented, showing that qualitative analysis has a great potential to deepen various dimensions of usability and functional and emotional interrelationships that are not by any means quantifiable.
Model-driven engineering of self-adaptive user interfaces
TLDR
A model-driven engineering approach for SAUIs is introduced with twofold contribution and new modeling languages for context-of-use (ContextML) and UI adaptation rules (AdaptML) are introduced.
webQDA — Qualitative data analysis software: Usability assessment
TLDR
An assessment of the usability of the qualitative data analysis software webQDA® (version 2.0), using the System Usability Scale (SUS) to assess its usability, indicates that the current version is “acceptable" in terms of usability.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 140 REFERENCES
Usability Engineering for the Adaptive Web
TLDR
This chapter discusses a usability engineering approach for the design and the evaluation of adaptive web-based systems, focusing on practical issues, and describes the evaluation methodologies following the temporal phases of evaluation, according to a user-centered approach.
Testing the usability of a personalized system: comparing the use of interviews, questionnaires and thinking-aloud
TLDR
Conurrent thinking-aloud turned out to be the best method to elicit comments on the perceived relevance of search results and a personalized search engine was evaluated to compare the use of interviews, questionnaires and concurrent thinking- aloud.
A survey of user-centered design practice
TLDR
Results show that cost-benefit tradeoffs are a key consideration in the adoption of UCD methods and that there is a major discrepancy between the commonly cited measures and the actually applied ones.
Workshop on User-Centred Design and Evaluation of Adaptive Systems
TLDR
This talk will first look at some special considerations that arise with user-adaptive systems such as the possibility of using relevant human experts either as judges of the appropriateness of the system’s adaptation or as wizards within the Wizard-of-Oz paradigm.
Eliciting Requirements for a Adaptive Decision SupportSystem through Structured User Interviews
TLDR
It is argued that scenarios in combination with structured interviews are not able to adequately identify adaptivity requirements due to reasons inherent to the method, such as the users’ trust and their ability to anticipate system functionality.
User-Centered Evaluation of Personalized Web Sites: What's Unique?
TLDR
This paper presents some of the questions and issues that must be addressed in user-centered evaluations of personalized Web sites, based on the experience in usability studies of a personalized e-commerce site.
Heuristic evaluation: Comparing ways of finding and reporting usability problems
Methods and techniques for the evaluation of user-adaptive systems
  • C. Gena
  • Computer Science
    The Knowledge Engineering Review
  • 2005
TLDR
The methodologies derived both from the evaluation of human–computer interaction systems and from information retrieval and information filtering systems are described by giving examples of the application of these methodologies in the user-adaptive systems.
Methodology for remote usability activities: A case study
TLDR
A survey of various types of remote usability activities and some of the types of tools that are being used is presented, and a case study in which this methodology was applied is described, that is, the design of the Design Solutions application for the IBM Customer Connect portal.
User interface evaluation in an iterative design process: a comparison of three techniques
  • Pam Savage
  • Computer Science
    CHI Conference Companion
  • 1996
TLDR
This paper compares the results of three different user interface evaluation techniques: expert reviews, user reviews, and interactive usability testing during the iterative design of a graphical user interface for an interactive multimedia messaging application.
...
1
2
3
4
5
...