Human-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization

  title={Human-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization},
  author={Mingming Fan and Xianyou Yang and Tsz Tung Yu and Vera Q. Z. Liao and Jian Zhao},
  journal={Proceedings of the ACM on Human-Computer Interaction},
  pages={1 - 32}
Analyzing usability test videos is arduous. Although recent research showed the promise of AI in assisting with such tasks, it remains largely unknown how AI should be designed to facilitate effective collaboration between user experience (UX) evaluators and AI. Inspired by the concepts of agency and work context in human and AI collaboration literature, we studied two corresponding design factors for AI-assisted UX evaluation: explanations and synchronization. Explanations allow AI to further… 

Figures and Tables from this paper

Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
This chapter begins with a high-level overview of the technical landscape of XAI algorithms, then selectively survey recent HCI works that take human-centered approaches to design, evaluate, and provide conceptual and methodological tools for XAI.
"It Feels Like Being Locked in A Cage": Understanding Blind or Low Vision Streamers' Perceptions of Content Curation Algorithms
Blind or low vision (BLV) people were recently reported to be live streamers on the online platforms that employed content curation algorithms. Recent research uncovered algorithm biases suppressing


CoUX: Collaborative Visual Analysis of Think-Aloud Usability Test Videos for Digital Interfaces
This work proposes a collaborative visual analytics tool, CoUX, to facilitate UX evaluators collectively reviewing think-aloud usability test videos of digital interfaces and indicates that CoUX is useful and effective in facilitating both problem identification and collaborative teamwork.
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
An algorithm-informed XAI question bank is developed in which user needs for explainability are represented as prototypical questions users might ask about the AI, and used as a study probe to identify gaps between current XAI algorithmic work and practices to create explainable AI products.
Concurrent Think-Aloud Verbalizations and Usability Problems
It is demonstrated that certain patterns of verbalizations are more telling of usability problems than others and that these patterns are robust to different types of test products, access to different kinds of information, and the presence or absence of a visualization of verbalization.
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
It is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud, and it is highly questionable to use a TA with 1evaluator as an authoritative statement about what problems an interface contains.
VisTA: Integrating Machine Intelligence with Visualization to Support the Investigation of Think-Aloud Sessions
Inspired by the recent research that shows subtle verbalization and speech patterns tend to occur when users encounter usability problems, this work takes the first step to design and evaluate an intelligent visual analytics tool that leverages such patterns to identify usability problem encounters and present them to UX practitioners to assist their analysis.
Automated Usability Evaluation of Virtual Reality Applications
  • Patrick Harms
  • Computer Science
    ACM Trans. Comput. Hum. Interact.
  • 2019
This article describes an approach, which is fully automated and does not ask users to perform predefined tasks in a fixed test setting, that works on recordings of the actual usage of a VR application from which it generates task trees and finds usability smells, i.e., user behavior indicating usability issues.
An exploratory study of co-located collaborative visual analytics around a tabletop display
An exploratory study of a system designed to support collaborative visual analysis tasks on a digital tabletop display and identifies eight types of collaboration styles that can be used to identify how closely people work together while problem solving.
What happened to remote usability testing?: an empirical study of three methods
Results from a systematic empirical comparison of three methods for remote usability testing and a conventional laboratory-based think-aloud method show that the remote synchronous method is virtually equivalent to the conventional method.
Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
An empirical study comparing the model learning outcomes, feedback content and experience with XAL, to that of traditional AL and coactive learning (providing the model’s prediction without explanation), and potential drawbacks–anchoring effect with the model judgment and additional cognitive workload.
Exploring Think-Alouds in Usability Testing: An International Survey
The reliability considerations underpinning the classic think-aloud approach are pragmatically balanced against the need to capture useful data in the time available, and future work is needed to explore the impact of divergent use of think- aloud instructions, practice tasks, and the use of demonstrations on test reliability.