Test and evaluation of a spoken dialogue system

@article{Gerbino1993TestAE,
  title={Test and evaluation of a spoken dialogue system},
  author={Elisabetta Gerbino and Paolo Baggia and Alberto Ciaramella and Claudio Rullent},
  journal={1993 IEEE International Conference on Acoustics, Speech, and Signal Processing},
  year={1993},
  volume={2},
  pages={135-138 vol.2}
}
The development of spoken dialogue systems (SDSs) requires the definition of evaluation metrics which can assess the performance of these systems at different levels and compare various SDSs. The authors present a first test, made with naive users, on an integrated dialogue system for telephone speech access to a remote data base. They describe the system architecture as well as the goals of the test, its features, the methodology used during the evaluation, and the results obtained. The SDS is… 
View on IEEE
doi.ieeecomputersociety.org
9 Citations
Background Citations
1

9 Citations

Citation Type

Citation Type

Parameters for Quantifying the Interaction with Spoken Dialogue Telephone Services
TLDR
A collection of parameters are presented which are now considered to be recommended by the International Telecommunication Union (ITU-T) for evaluating telephone-based spoken dialogue services and show that they still may be used for predicting quality with PARADISE-style regression models.
Evaluating Interactions with Spoken Dialogue Telephone Services
TLDR
This chapter presents standardised methods for both measurement approaches and an initial evaluation study in subjective evaluation experiments shows that the parameters correlate only weakly with subjective judgements; thus, both types of evaluation provide complementary types of information.
Spontaneous speech dialogue system TOSBURG II and its evaluation
Assessment and Evaluation of Speech-Based Interactive Systems: From Manual Annotation to Automatic Usability Evaluation
TLDR
Both individual component assessment and entire system evaluation are worth being considered here, depending on the question which shall be answered by the assessment or evaluation.
Application Domain, Human Factors, and Dialogue
TLDR
This chapter concentrates on theUse of application domain knowledge, the user interface, and the use of dialogue to improve the robustness of an application using speech to build application prototypes and to reduce development costs.
The Philips automatic train timetable information system
Experience with the Philips automatic train timetable information system
TLDR
An automatic system for train timetable information over the telephone that provides accurate connections between 1200 German cities that is made available to the general public, both to gather speech data and to evaluate its performance.
Qualität von Sprachdialogsystemen
Nachdem in den vergangenen beiden Kapiteln Systeme zur technischen Unterstutzung zwischenmenschlicher Kommunikation betrachtet wurden, befassen wir uns in diesem und dem folgenden Kapitel mit der…
Parameters describing the interaction with multimodal dialogue systems
  • Medicine
  • 2011

References

SHOWING 1-8 OF 8 REFERENCES
Real-time linguistic analysis for continuous speech understanding
TLDR
This paper describes the approach followed in the development of the linguistic processor of the continuous speech dialog system implemented at the labs, and results are discussed, as obtained from an implementation of the system on a Sun SparcStation 1 using the C language.
Developing an Evaluation Methodology for Spoken Language Systems
TLDR
An overview of the process that was followed in creating a meaningful evaluation mechanism is given, the current mechanism is described, and some directions for future development are presented.
Collection of Spontaneous Speech for the ATIS Domain and Comparative Analyses of Data Collected at MIT and TI
TLDR
This paper documents the data collection process, and makes some comparative analyses of the data with those collected at Texas Instruments, in the ATIS domain.
A Proposal for Incremental Dialogue Evaluation
TLDR
The goal is to develop incremental ways to evaluate dialogue processing, not just going from Class D1 (dialogue pairs) to Class D2 ( Dialogue triples), but measuring aspects of dialogue processing other than length.
Comparison of discrete and continuous HMMs in a CSR task over the telephone
  • L. Fissore, P. Laface, G. Micca
  • Computer Science
    [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing
  • 1991
Attention is given to a comparison of the performance of discrete and continuous density hidden Markov models (DDHMMs and CDHMMs) on a 786-word E-mail inquiry task performed by the…
Partial parsing as a robust parsing strategy
  • P. Baggia, C. Rullent
  • Computer Science
    1993 IEEE International Conference on Acoustics, Speech, and Signal Processing
  • 1993
TLDR
A robust parsing strategy where partial parsing is seen not as a back-up strategy, but as the normal mode of operation of the parser, which makes it possible to increase robustness to spontaneous speech and to reduce the effect of limitations in the syntactic/semantic coverage of the grammar.
A man-machine dialogue system for speech access to e-mail information using the telephone: implementation and first results
Interpretation of context-dependent utterances in man-machine dialogue