Evaluating and Informing the Design of Chatbots

@article{Jain2018EvaluatingAI,
  title={Evaluating and Informing the Design of Chatbots},
  author={Mohit Jain and Pratyush Kumar and Ramachandra Kota and Shwetak N. Patel},
  journal={Proceedings of the 2018 Designing Interactive Systems Conference},
  year={2018}
}
Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with… 

Figures and Tables from this paper

User Expectations of Conversational Chatbots Based on Online Reviews
TLDR
Drawing from reviews of chatbots posted on Google Play, user experience and expectations of these agents are explored in a mixed-method study, revealing which social qualities of chatbot are the most significant for user satisfaction.
Informing the Design of a News Chatbot
TLDR
A qualitative user study to better understand users' attitudes towards news chatbots, and derives several design guidelines on effectiveness, informativeness, efficiency, humanization, and facility, suggesting developments to news chatbot that may better serve users' needs.
Conversational Survey Chatbot: User Experience and Perception
The aim of the paper is to explore two directions, one is the use of conversational chatbots for the purpose of data collection, and the other is the analysis of user experience in the usage of
Trends & Methods in Chatbot Evaluation
TLDR
A review of current techniques and trends in chatbot evaluation, which identifies a clear trend towards evaluating the efficiency of chatbots in many recent papers, which is linked to the growing popularity of task-based chatbots that are currently being deployed in many business contexts.
Chatbots Language Design: The Influence of Language Variation on User Experience with Tourist Assistant Chatbots
TLDR
The results show that register characteristics are strong predictors of user’s preferences, which points to the needs of designing chatbots with register-appropriate language to improve acceptance and users’ perceptions of chatbot interactions.
BigBlueBot: teaching strategies for successful human-agent interactions
TLDR
This work designs a fun learning experience with several goals: explaining how chatbots work by mapping utterances to a set of intents, teaching strategies for avoiding conversational breakdowns, and increasing desire to use chatbots by creating feelings of empathy toward them.
How should my chatbot interact ?
TLDR
It is argued that chatbots should be enriched with social characteristics that are coherent with users’ expectations, ultimately avoiding frustration and dissatisfaction.
Do Multilingual Users Prefer Chat-bots that Code-mix? Let's Nudge and Find Out!
TLDR
A mixed-method user-study examines how conversational agents, that code-mix and reciprocate the users' mixing choices over multiple conversation turns, are evaluated and perceived by bilingual users, and finds that self-reported language proficiency is the strongest predictor of user preferences.
(Non-)Interacting with conversational agents: perceptions and motivations of using chatbots and voice assistants
TLDR
How users currently perceive CAs is revealed and quality criteria that could inform their future design are identified to contribute to the field by extending these guidelines from an end-user's perspective.
Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues
TLDR
A framework for qualitative analysis of chatbot dialogues in the customer service domain is presented and it is shown how it may provide insight into key drivers of user experience, including response relevance and dialogue helpfulness.
...
...

References

SHOWING 1-10 OF 46 REFERENCES
Convey: Exploring the Use of a Context View for Chatbots
TLDR
Convey (CONtext View) is proposed, a window added to the chatbot interface, displaying the conversational context and providing interactions with the context values, and a discussion of the design implications offered by Convey.
All Work and No Play? Conversations with a Question-and-Answer Chatbot in the Wild
TLDR
By studying a field deployment of a Human Resource chatbot, data is reported on users' interest areas in conversational interactions to inform the development of CAs, and rich signals in Conversational interactions are highlighted for inferring user satisfaction with the instrumental usage and playful interactions with the agent.
How Do You Want Your Chatbot? An Exploratory Wizard-of-Oz Study with Young, Urban Indians
TLDR
Overall the results show that users wanted a chatbot like Maya, who could add value to their life while being a friend, by making useful recommendations, but they also wanted preferred traits of Ada and Evi infused into Maya.
All Work and No Play?
TLDR
By studying a field deployment of a Human Resource chatbot, data is reported on users' interest areas in conversational interactions to inform the development of CAs, and rich signals in Conversational interactions are highlighted for inferring user satisfaction with the instrumental usage and playful interactions with the agent.
"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents
TLDR
This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use, and finds user expectations dramatically out of step with the operation of the systems.
What would you Ask a conversational Agent? Observations of Human-Agent Dialogues in a Museum Setting
TLDR
This paper explores issues through an analysis of a corpus that has been collected since December 2006, from interactions with the virtual character Sgt Blackwell at the Cooper Hewitt Museum in New York, which includes 82 hierarchical categories of user utterances, as well as specific observations on user preferences and behaviors drawn from interaction with Blackwell.
Talking Eye: autonomous creature as accomplice for human
TLDR
An autonomous creature called Talking Eyes is presented, which has the advantage of a property of chatting, which helps build autonomous creatures with a mechanism to promote a more consensual feeling through its interaction with humans.
Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue
TLDR
A framework for evaluating the dialogue competence of artificial agents involved in complex and underspecified tasks when conversing with people is presented and a multi-part coding scheme is proposed that provides a qualitative analysis of human utterances, and rates the appropriateness of the agent's responses to these utterances.
Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues
This paper summarizes discussions at the multidisciplinary forum held at the University of Oxford on 26 October 2007 entitled Artificial Companions in Society: Perspectives on the Present and Future,
Advances in natural multimodal dialogue systems
TLDR
The aim of this monograph is to provide a Discussion of the Foundations of Multimodal Interactivity Engineering and its Applications in Speech, Gestures, and Object Manipulation, as well as some of the approaches used in this work.
...
...