A Simple Model for Subject Behavior in Subjective Experiments

@article{Li2020ASM,
  title={A Simple Model for Subject Behavior in Subjective Experiments},
  author={Zhi Li and Christos George Bampis and Lucjan Janowski and Ioannis Katsavounidis},
  journal={ArXiv},
  year={2020},
  volume={abs/2004.02067}
}
In a subjective experiment to evaluate the perceptual audiovisual quality of multimedia and television services, raw opinion scores offered by subjects are often noisy and unreliable. Recommendations such as ITU-R BT.500, ITU-T P.910 and ITU-T P.913 standardize post-processing procedures to clean up the raw opinion scores, using techniques such as subject outlier rejection and bias removal. In this paper, we analyze the prior standardized techniques to demonstrate their weaknesses. As an… 

Figures from this paper

Generalised Score Distribution: A Two-Parameter Discrete Distribution Accurately Describing Responses from Quality of Experience Subjective Experiments

The proposed Generalised Score Distribution (GSD) properly describes response distributions observed in typical MQA experiments and it is indicated that the GSD outperforms the approach based on sample empirical distribution when it comes to bootstrapping.

Subjective Assessment of High Dynamic Range Videos Under Different Ambient Conditions

High Dynamic Range (HDR) videos can represent a much greater range of brightness and color than Standard Dynamic Range (SDR) videos and are rapidly becoming an industry standard. HDR videos have more

Telepresence Video Quality Assessment

This work has created a first-of-a-kind online video quality prediction framework for live streaming, using a multi-modal learning framework with separate pathways to visual and audio quality predictions, able to provide accurate quality predictions at the patch, frame, clip, and audiovisual levels.

Subjective and Objective Quality Assessment of High-Motion Sports Videos at Low-Bitrates

The first large-scale study of medium and low-bitrate videos from live sports for two codecs (Elemental AVC and HEVC) and the Amazon Prime Video Low-Bitrate Sports (APV LBS) dataset is conducted and an extensive evaluation of objective Video Quality Assessment (VQA) algorithms is conducted.

Generalised Score Distribution: Underdispersed Continuation of the Beta-Binomial Distribution

A class of discrete probability distributions contains distributions with limited support. A typical example is some variant of a Likert scale, with response mapped to either the { 1 , 2 , . . . , 5

Systematic Analysis of Experiment Precision Measures and Methods for Experiments Comparison

The main idea of this paper is to propose a framework for comparing subjective experiments in the Multimedia Quality Assessment based on appropriate experiment precision measures and three related experiment precision comparison methods, which will help compare different subjective experiment methodologies.

Visual Quality of 3D Meshes With Diffuse Colors in Virtual Reality: Subjective and Objective Evaluation

This work proposes the first metric for quality assessment of 3D meshes with diffuse colors, which works entirely on the mesh domain and incorporates perceptually-relevant curvature-based and color-based features.

A Probabilistic Graphical Model for Analyzing the Subjective Visual Quality Assessment Data from Crowdsourcing

A probabilistic graphical annotation model to infer the underlying ground truth and discovering the annotator's behavior is proposed, which shows superior performance than the other state-of-the-art models in terms of accuracy and robustness.

Strategy for Boosting Pair Comparison and Improving Quality Assessment Accuracy

A generic model is employed to bridge the pair comparison data and ACR data, where the variance term could be recovered and the obtained information is more complete, and a fusion strategy to boost pair comparisons by utilizing the ACR results as initialization information is proposed.

References

SHOWING 1-10 OF 18 REFERENCES

ITU-T coded-speech database

  • Technical Report Series P: Telephone Transmission Quality, Telephone Installations, Local Line Networks, ITU-T, Feb.
  • 1998

The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study

This paper addresses the question of determining the most suitable way to conduct audiovisual subjective testing on a wide range of audiovISual quality, and analyses show that the results of experiments done in pristine, laboratory environments are highly representative of those devices in actual use, in a typical user environment.

Recover Subjective Quality Scores from Noisy Measurements

  • Zhi LiC. Bampis
  • Computer Science
    2017 Data Compression Conference (DCC)
  • 2017
This paper proposes a new approach to recover subjective quality scores from noisy raw measurements, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together.

Generalized Score Distribution

The proposed Generalized Score Distribution covers all spreads of the answers, from very small, given by the Bernoulli distribution, to the maximum given by a Beta Binomial distribution and it is shown that GSD correctly describes subjective experiments scores from video quality evaluations with probability of 99.7%.

ITS4S2: An Image Quality Dataset With Unrepeated Images From Consumer Cameras

This technical memorandum provides technical details for the image quality subjective experiment its4s2, to train no reference metrics that track the imagequality of consumer grade cameras, to understand spatial characteristics of moving objects that have complex interactions with video codecs, such as water and confetti.

AGH/NTIA: A Video Quality Subjective Test with Repeated Sequences

This report provides full technical details for the video quality subjective test AGH/NTIA. Analyses of this dataset appear in separate publications. The purpose of this document is to provide design

ITS4S: A Video Quality Dataset with Four-Second Unrepeated Scenes

This report describes the video quality subjective test its4s, including the experiment design and footage attribution to provide insights into the optimal experiment designs for training no-reference metrics and to understand the impact of original video quality on mean opinion scores (MOS).

The Accuracy of Subjects in a Quality Experiment: A Theoretical Subject Model

A theoretical subject model is proposed that indicates that the accuracy and accuracy of most subjective analyses can be improved if the subject scores are normalized by removing subject bias and stimulus scoring difficulty.

Subject bias: Introducing a theoretical user model

  • L. JanowskiM. Pinson
  • Economics
    2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX)
  • 2014
It is concluded that bias acts like a random variable within ratings, and it is advantageous to remove subject bias when analyzing mean opinion score.

Elements of Information Theory

The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.