QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS

  title={QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS},
  author={Tobias Hossfeld and Poul Einar Heegaard and Mart{\'i}n Varela and Sebastian M{\"o}ller},
  journal={Quality and User Experience},
While Quality of Experience (QoE) has advanced very significantly as a field in recent years, the methods used for analyzing it have not always kept pace. When QoE is studied, measured or estimated, practically all the literature deals with the so-called Mean Opinion Score (MOS). The MOS provides a simple scalar value for QoE, but it has several limitations, some of which are made clear in its name: for many applications, just having a mean value is not sufficient. For service and content… 

Betas: Deriving quantiles from MOS-QoS relations of IQX models for QoE management

The derivation of quantiles from existing MOS-QoS relations allows to reuse existing subjective MOS results and MOS models without rerunning the experiments and a practical guideline for the computation of the quantiles is provided.

Deriving QoE in systems: from fundamental relationships to a QoE-based Service-level Quality Index

A more generalized view on quantifying the quality of systems by defining a QoE-based Service-level Quality Index, which exploits the fact that quality can be seen as a proxy measure for utility to provide a generic framework to quantify the overall utility of a service delivered by a system.

Embracing uncertainty: A probabilistic view of HTTP video quality

  • M. VarelaToni Mäki
  • Computer Science
    2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM)
  • 2017
This paper presents a first step towards working with all the information available in an explicit way, rather than averaging it away, in the context of constructing layered quality models for HTTP video streaming (using Dynamic Adaptive HTTP Streaming — DASH), mapping network-level QoS measurements to probability distributions of different MOS values for a given set of conditions.

From QoS Distributions to QoE Distributions: a System's Perspective

This paper shows how to approximate user rating distributions given a QoS-to-MOS mapping function and second order statistics, and provides two examples to illustrate this process.

Active Learning for Crowdsourced QoE Modeling

It is suggested that simply simulating the sample acquisition order, which is widely adopted in previous active learning literature  [2]–[5], is not sufficient for multimedia quality assessment tasks.

A Quality-of-Experience Database for Adaptive Video Streaming

A new database named Waterloo Streaming QoE Database III is established, so far the largest and most realistic of its kind, and consists of a total of 450 streaming videos created from diverse source content and diverse distortion patterns, with six adaptation algorithms of diverse characteristics under 13 representative network conditions.

No silver bullet: QoE metrics, QoE fairness, and user diversity in the context of QoE management

This paper discusses how the choice of measures used, the importance of fairness, and how the variations between users can affect the optimal QoE management choices for service providers.

Formal Definition of QoE Metrics

This technical report formally defines the QoE metrics which are introduced and discussed in the article "QoE Beyond the MOS: An In-Depth Look at QoE via Better Metrics and their Relation to MOS" by

Statistical methods and models based on quality of experience distributions

The goal is to demonstrate the fundamental advantages of considering QoE distributions over MOS-based evaluations if the underlying rating data is ordinal in nature.

Confidence Interval Estimators for MOS Values

It is shown that the provided CI estimator works very well in practice for MOS interval estimators, while the commonly used studentized CIs suffer from a positive outlier ratio, i.e., CIs beyond the bounds of the rating scale.



QoE beyond the MOS: Added value using quantiles and distributions

Different statistical measures to express important aspects of QoE beyond MOS like user diversity, uncertainty of user rating distributions, ratio of dissatisfied users are proposed.

SOS: The MOS is not enough!

The main benefit of the SOS hypothesis is that it allows for a compact, yet still comprehensive statistical summary of subjective user tests and supports checking the reliability of test result data sets as well as their comparability across different QoE studies.

Metrics for the Internet Age: Quality of Experience and Quality of Business

A QoBiz evaluation framework is introduced, responding to the emergence of three types of services that impact quantitative evaluation: business to consumer services, business to business services, and service utility through service providers.

The memory effect and its implications on Web QoE modeling

Three different QoE models are proposed which consider the implications of the memory effect and imply the required extensions of the basic models, and are described with a) support vector machines, b) iterative exponential regressions, and c) two-dimensional hidden Markov models.

Formal Definition of QoE Metrics

This technical report formally defines the QoE metrics which are introduced and discussed in the article "QoE Beyond the MOS: An In-Depth Look at QoE via Better Metrics and their Relation to MOS" by

Acceptability and Quality of Experience in over the top video

The experimental results indicate that TQ and OX have a strong impact on acceptability for videos containing impairment and failure events during lifecycle, thus affecting Accessibility, Retainability and Integrity.

Towards a comprehensive framework for QOE and user behavior modelling

  • P. ReichlS. Egger A. Asrese
  • Computer Science
    2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX)
  • 2015
A comprehensive QoE and user behavior model is presented providing a framework which allows joining a multitude of existing modeling approaches under the perspectives of service provider benefit, user well-being and technical system performance.

Of packets and people: a user-centered approach to quality of service

  • A. BouchM. SasseH. deMeer
  • Computer Science
    2000 Eighth International Workshop on Quality of Service. IWQoS 2000 (Cat. No.00EX400)
  • 2000
Cognitive and perceptive conditioning of users and applications in a situated setting is investigated, and methodological foundations are laid for investigating user-centered QoS.

Network quality differentiation: Regional effects, market entrance, and empirical testability

The integral empirical Willingness-To-Pay (WTP) data for the case of online video services is collected and insights and updated methodological guidance on conducting future WTP studies are given.

Modeling subjective tests of quality of experience with a Generalized Linear Model

  • L. JanowskiZ. Papir
  • Computer Science
    2009 International Workshop on Quality of Multimedia Experience
  • 2009
Monitoring a QoE level with a GLZ model provides to operators an effective tool towards a mid- and long-term adaptation of both network and/or application parameters in order to follow users' experience and expectations.