Real User Evaluation of Spoken Dialogue Systems Using Amazon Mechanical Turk

@inproceedings{Jurccek2011RealUE,
  title={Real User Evaluation of Spoken Dialogue Systems Using Amazon Mechanical Turk},
  author={Filip Jurc{\'i}cek and Simon Keizer and Milica Gasic and François Mairesse and Blaise Thomson and Kai Yu and Steve J. Young},
  booktitle={INTERSPEECH},
  year={2011}
}
This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting… CONTINUE READING
Highly Cited
This paper has 81 citations. REVIEW CITATIONS

3 Figures & Tables

Topics

Statistics

0102020112012201320142015201620172018
Citations per Year

82 Citations

Semantic Scholar estimates that this publication has 82 citations based on the available data.

See our FAQ for additional information.