Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems

  title={Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems},
  author={Jason Ingyu Choi and Ali Ahmadvand and Eugene Agichtein},
  journal={Proceedings of the 28th ACM International Conference on Information and Knowledge Management},
Predicting user satisfaction in conversational systems has become critical, as spoken conversational assistants operate in increasingly complex domains. Online satisfaction prediction (i.e., predicting satisfaction of the user with the system after each turn) could be used as a new proxy for implicit user feedback, and offers promising opportunities to create more responsive and effective conversational agents, which adapt to the user's engagement with the agent. To accomplish this goal, we… 

Figures and Tables from this paper

User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems

This paper proposes a novel framework, namely USDA, to incorporate the sequential dynamics of dialogue acts for predicting user satisfaction, by jointly learning User Satisfaction Estimation and Dialogue Act Recognition tasks.

ConvSearch: A Open-Domain Conversational Search Behavior Dataset

A novel conversational search platform is developed to collect dialogue contents, annotate dialogue quality and candidate search results and record agent search behaviors and provide the intent labels of each search behavior iteration to support intent understanding related researches.

How Am I Doing?: Evaluating Conversational Search Systems Offline

This work proposes a framework for offline evaluation of conversational search, which includes a methodology for creating test collections with relevance judgments, an evaluation measure based on a user interaction model, and an approach to collecting user interaction data to train the model.

Meta-evaluation of Conversational Search Evaluation Metrics

This work establishes the most comprehensive meta-evaluation for conversational search metrics considering all three perspectives, and proves that adapted session-based evaluation metrics can be used to measure multi-turn Conversational search, achieving moderate concordance with user satisfaction.

What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts

A novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality and is the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable.

Turn-Level User Satisfaction Estimation in E-commerce Customer Service

This paper formalizes the turn-level satisfaction estimation as a reinforcement learning problem, in which the model can be optimized with only session-levelatisfaction labels, and shows that the proposed method outperforms all the baseline models.

Quantifying the Effects of Prosody Modulation on User Engagement and Satisfaction in Conversational Systems

A large-scale empirical study measures the effects of prosodic modulation on user behavior and engagement across multiple conversation domains, both immediately aftereach turn, and at the overall conversation level, and indicates that the prosody modulation significantly increases both immediate and overall user satisfaction.

Would you Like to Talk about Sports Now?: Towards Contextual Topic Suggestion for Open-Domain Conversational Agents

Three possible approaches for a personalized, contextual topic suggestion for open-domain conversations are explored and the results are promising: the CTS-Seq model suggests topics with 23% higher accuracy than the baseline, and incorporating collaborative filtering signals into a hybrid CTS -Seq-CF model further improves recommendation accuracy by 12%.

User Intent Inference for Web Search and Conversational Agents

This thesis work proposed novel models to incorporate entity information and conversation-context clues to predict both topic and intent of the user's utterances to improve performance of various tasks such as natural language understanding, query scoping, query suggestion, and ranking, resulting in an enriched user experience.

HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

This work proposes HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem, and improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.



On Evaluating and Comparing Open Domain Dialog Systems

This paper proposes a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement, and believes that this work is a step towards an automatic evaluation process for conversational AIs.

Conversational AI: The Science Behind the Alexa Prize

The advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI are outlined.

Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

On the PersonaChat chit-chat dataset with over 131k training examples, it is found that learning from dialogue with a self-feeding chatbot significantly improves performance, regardless of the amount of traditional supervision.

Learning Personalized End-to-End Goal-Oriented Dialog

A personalized end-to-end model in an attempt to leverage personalization in goal-oriented dialogs that achieves qualitative performance improvements over state-of-the-art methods and outperforms other approaches in terms of task completion rate and user satisfaction.

IrisBot : An Open-Domain Conversational Bot for Personalized Information Access

We describe IrisBot, a conversational agent that aims to help a customer be informed about the world around them, while being entertained and engaged. Our bot attempts to incorporate real-time

Wizard of Wikipedia: Knowledge-Powered Conversational agents

The best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while a new benchmark allows for measuring further improvements in this important research direction.

End-to-End Reinforcement Learning of Dialogue Agents for Information Access

This paper proposes KB-InfoBot -- a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agents typically need

End-to-end joint learning of natural language understanding and dialogue manager

Experiments show that the proposed model significantly outperforms the state-of-the-art pipeline models for both NLU and SAP, which indicates that the joint model is capable of mitigating the affects of noisy NLU outputs, and NLU model can be refined by error flows backpropagating from the extra supervised signals of system actions.

ConCET: Entity-Aware Topic Classification for Open-Domain Conversational Agents

ConCET: a Concurrent Entity-aware conversational Topic classifier, which incorporates entity type information together with the utterance content features, and a simple and effective method for generating synthetic training data, to augment the typically limited amounts of labeled training data.

From Eliza to XiaoIce: challenges and opportunities with social chatbots

The success metric for social chatbots is defined as conversation-turns per session (CPS), and it is shown how XiaoIce can dynamically recognize emotion and engage the user throughout long conversations with appropriate interpersonal responses.