• Corpus ID: 6656629

Evaluating Quality of Chatbots and Intelligent Conversational Agents

  title={Evaluating Quality of Chatbots and Intelligent Conversational Agents},
  author={Nicole M. Radziwill and Morgan C. Benton},
Chatbots are one class of intelligent, conversational software agents activated by natural language input (which can be in the form of text, voice, or both). They provide conversational output in response, and if commanded, can sometimes also execute tasks. Although chatbot technologies have existed since the 1960s and have influenced user interface development in games since the early 1980s, chatbots are now easier to train and implement. This is due to plentiful open source code, widely… 

Figures and Tables from this paper

Trends & Methods in Chatbot Evaluation
A review of current techniques and trends in chatbot evaluation, which identifies a clear trend towards evaluating the efficiency of chatbots in many recent papers, which is linked to the growing popularity of task-based chatbots that are currently being deployed in many business contexts.
Chatbots Explain Themselves: Designers' Strategies for Conveying Chatbot Features to Users
This work uses the Semiotic Inspection Method to identify a series of strategies used by the analyzed chatbots for conveying their features to users, and discusses the use of these strategies, as well as challenges for designing such interfaces and limitations of using SIM on them.
KINO: an approach for rule-based chatbot development, monitoring and evaluation
KINO’s architecture is presented, along with challenges and proposed solutions described as a reference for the development of other chatbots, and issues that allow future maintenance and possible improvements of KINO are shown.
Helping Chatbots To Better Understand User Requests Efficiently Using Human Computation
A methodology to generate high quality training data is proposed, with which, chatbot’s Natural Language Understanding (NLU) model can be trained, making a chatbot capable of handling user requests efficiently at run time and a methodology to estimate the reliability of black box NLU models based on the confidence threshold of their prediction functionality.
AIML and Sequence-to-Sequence Models to Build Artificial Intelligence Chatbots: Insights from a Comparative Analysis
This study critically compares the Artificial Intelligence Mark-up Language (AIML), and Sequence-to-Sequence models for building chatbots and showed that the AIML chatbot ensured better user satisfaction, and task completion rate, while the Sequence- to- Sequence model had better information retrieval rate.
Limitations of Existing Chatbot with Analytical Survey to Enhance the Functionality Using Emerging Technology
The main aim of this research is to find a better way to the problems that can be solved and the user can use the chatbot flawlessly and smoothly.
Do people want to message Chatbots?: developing and comparing the usability of a conversational vs. menu-based Chatbot in context of new hire onboarding
The results indicate that users preferred a menu-based over a conversational chatbot experience due to its greater ease of use, less likelihood for errors, convenience of graphical user interface elements, and suitability for scenarios where information needs to be provided rather than requested.
Prototyping a Chatbot for Student Supervision in a Pre-Registration Process
This research has proved that the combination of keyword spotting technique for the Language Understanding component, Finite-State Transducer (FST) for the Dialogue Management, rulebased keyword matching for language generation, and the system-in-the-loop paradigm for system validation can produce an efficient chatbot.
Generalised Framework for Automated Conversational Agent Design via QFD
This paper explores some product design ideas such as Analytic Hierarchy Process (AHP) and Quality Function Deployment (QFD) drawn from industrial engineering literature for chatbot development.


A Black-box Approach for Response Quality Evaluation of Conversational Agent Systems
A blackbox approach is proposed using observation, classification scheme and a scoring mechanism to assess and rank three example systems, AnswerBus, Start and AINI, to demonstrate the challenges in evaluating systems of different nature.
The linguistic accuracy of chatbots: usability from an ESL perspective
Abstract This paper reports on the linguistic accuracy of five renowned “chatbots,” with an evaluator (an ESL teacher) chatting with each chatbot for about three hours. The chatting consisted of a
Toward the implementation of a topic specific dialogue based natural language chatbot as an undergraduate advisor
Model the Information Repository by a connected graph where the nodes contain information and links interrelates the information nodes and suggests that topic specific dialogue coupled with conversational knowledge yield the maximum dialogue session than the general conversational dialogue.
Conversational Interfaces: Past and Present
This chapter reviews developments in spoken dialog systems, VUI, embodied conversational agents, social robots, and chatbots, and outlines findings and achievements from this work that will be important for the next generation of conversational interfaces.
'Realness' in Chatbots: Establishing Quantifiable Criteria
The aim of this research is to generate measurable evaluation criteria acceptable to chatbot users, resulting in four subscales with strong reliability which discriminated well between the two categories of chatbots.
Mobile conversational commerce: messenger chatbots as the next interface between businesses and consumers
Nowadays, businesses are slowly starting to deploy mobile messenger chatbots as a new method of communication with its customers. Due to the subject’s infancy and lack of research on the subject, the
Even good bots fight: The case of Wikipedia
Analysis of the interactions between bots that edit articles on Wikipedia suggests that even relatively “dumb” bots may give rise to complex interactions, and this carries important implications for Artificial Intelligence research.
Even Good Bots Fight
It is found that, although Wikipedia bots are intended to support the encyclopedia, they often undo each other’s edits and these sterile “fights” may sometimes continue for years.
Automation, Algorithms, and Politics| Talking to Bots: Symbiotic Agency and the Case of Tay
In 2016, Microsoft launched Tay, an experimental artificial intelligence chat bot. Learning from interactions with Twitter users, Tay was shut down after one day because of its obscene and