• Corpus ID: 233004444

MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation

@article{Ye2021MultiWOZ2A,
  title={MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation},
  author={Fanghua Ye and Jarana Manotumruksa and Emine Yilmaz},
  journal={ArXiv},
  year={2021},
  volume={abs/2104.00773}
}
The MultiWOZ 2.0 dataset was released in 2018. It consists of more than 10,000 taskoriented dialogues spanning 7 domains, and has greatly stimulated the research of taskoriented dialogue systems. However, there is substantial noise in the state annotations, which hinders a proper evaluation of dialogue state tracking models. To tackle this issue, massive efforts have been devoted to correcting the annotations, resulting in 3 improved versions of this dataset (i.e., MultiWOZ 2.12.3). Even so… 

Figures and Tables from this paper

Description-Driven Task-Oriented Dialog Modeling
TLDR
This paper proposes that schemata should be modified by replacing names or notations entirely with natural language descriptions, and shows that a language description-driven system exhibits better understanding of task specifications, higher performance on state tracking, improved data efficiency, and effective zero-shot transfer to unseen tasks.
Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking
TLDR
It is hypothesized that dialogue summaries are essentially unstructured dialogue states; hence, it is proposed to reformulate dialogue state tracking as a dialogue summarization problem, and it is discovered that the naturalness of the summary templates plays a key role for successful training.
In-Context Learning for Few-Shot Dialogue State Tracking
TLDR
This work proposes an in-context (IC) learning framework for few-shot dialogue state tracking (DST), where a large pre-trained language model (LM) takes a test instance and a few annotated examples as input, and directly decodes the dialogue states without any parameter updates.
What Did You Say? Task-Oriented Dialog Datasets Are Not Conversational!?
TLDR
This work outlines a taxonomy of conversational and contextual effects, which is used to examine MULTIWOZ, SGD and SMCALFLOW, among the most recent and widely used task-oriented dialog datasets, and outlines desiderata for truly conversational dialog datasets.
ASSIST: Towards Label Noise-Robust Dialogue State Tracking
TLDR
This paper proposes a general framework, named ASSIST (lAbel noiSerobuSt dIalogue State Tracking), to train DST models robustly from noisy labels, and shows the validity of ASSIST theoretically.
A Chit-Chats Enhanced Task-Oriented Dialogue Corpora for Fuse-Motive Conversation Systems
TLDR
This work releases a multi-turn dialogues dataset called Chinese ChatEnhanced-Task (CCET) and proposes a line of fuse-motive dialogues formalization approach, along with several evaluation metrics for TOD sessions that are integrated by CC utterances.
A Few-Shot Semantic Parser for Wizard-of-Oz Dialogues with the Precise ThingTalk Representation
TLDR
A new dialogue representation and a sample-efficient methodology that can predict precise dialogue states in WOZ conversations are proposed and extended the ThingTalk representation to capture all information an agent needs to respond properly.
Robust Dialogue State Tracking with Weak Supervision and Sparse Data
TLDR
A training strategy to build extractive DST models without the need for fine-grained manual span labels, and a new model architecture with a unified encoder that supports value as well as slot independence by leveraging the attention mechanism.
ViWOZ: A Multi-Domain Task-Oriented Dialogue Systems Dataset For Low-resource Language
TLDR
ViWOZ is the first multi-turn, multi-domain tasked oriented dataset in Vietnamese, a low-resource language, and provides a comprehensive benchmark of both modular and end-to-end models in lowresource language scenarios.
AllWOZ: Towards Multilingual Task-Oriented Dialog Systems for All
TLDR
AllWOZ is presented, a multilingual multi-domain task-oriented customer service dialog dataset covering eight languages: English, Mandarin, Korean, Vietnamese, Hindi, French, Portuguese, and Thai, and a benchmark for this multilingual dataset is created by applying mT5 (Xue et al., 2021) with meta-learning.
...
1
2
...

References

SHOWING 1-10 OF 25 REFERENCES
MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines
TLDR
This work identifies and fixes dialogue state annotation errors across 17.3% of the utterances on top of MultiWOZ 2.1, and redefines the ontology by disallowing vocabularies of slots with a large number of possible values to help avoid annotation errors.
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
TLDR
The Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics is introduced, at a size of 10k dialogues, at least one order of magnitude larger than all previous annotated task-oriented corpora.
RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modeling
TLDR
RiSAWOZ is a large-scale multi-domain Chinese Wizard-of-Oz dataset with Rich Semantic Annotations, which contains 11.2K human-to-human multi-turn semantically annotated dialogues, with more than 150K utterances spanning over 12 domains, which is larger than all previous annotated H2H conversational datasets.
CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset
TLDR
The large size and rich annotation of CrossWOZ make it suitable to investigate a variety of tasks in cross-domain dialogue modeling, such as dialogue state tracking, policy learning, user simulation, etc.
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
TLDR
A Transferable Dialogue State Generator (TRADE) that generates dialogue states from utterances using copy mechanism, facilitating transfer when predicting (domain, slot, value) triplets not encountered during training.
Slot Self-Attentive Dialogue State Tracking
TLDR
This paper proposes a slot self-attention mechanism that can learn the slot correlations automatically, and achieves state-of-the-art performance on both datasets, verifying the necessity and effectiveness of taking slot correlations into consideration.
Slot Attention with Value Normalization for Multi-domain Dialogue State Tracking
TLDR
A new architecture to cleverly exploit ontology, which consists of Slot Attention (SA) and Value Normalization (VN), referred to as SAVN, is proposed, which achieves the state-of-the-art joint accuracy and evaluation results show that even if only 30% ontology is used, VN can also contribute to the model.
Frames: a corpus for adding memory to goal-oriented dialogue systems
TLDR
A rule-based baseline is proposed and the frame tracking task is proposed, which consists of keeping track of different semantic frames throughout each dialogue, and the task is analysed through this baseline.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
TLDR
This work proposes a novel Neural Belief Tracking (NBT) framework which overcomes past limitations, matching the performance of state-of-the-art models which rely on hand-crafted semantic lexicons and outperforming them when such lexicons are not provided.
A Network-based End-to-End Trainable Task-oriented Dialogue System
TLDR
This work introduces a neural network-based text-in, text-out end-to-end trainable goal-oriented dialogue system along with a new way of collecting dialogue data based on a novel pipe-lined Wizard-of-Oz framework that can converse with human subjects naturally whilst helping them to accomplish tasks in a restaurant search domain.
...
1
2
3
...