Quality of Experience Models for Multimedia Streaming


Understanding how quality is perceived by the viewers of multimedia streaming services is essential for efficient management of those services. Quality of Experience (QoE) is a subjective metric that quantifies the perceived quality and therefore is crucial in the process of optimizing the tradeoff between quality and resources. However, accurate estimation of QoE usually entails cumbersome subjective studies that are long and expensive to execute. This paper presents a QoE estimation methodology for developing Machine Learning prediction models based on initial restricted-size subjective tests. Experimental results on subjective data from streaming multimedia tests show that the Machine Learning models outperform other statistical methods achieving accuracy greater than 90%. These models are suitable for real-time use due to their small computational complexity. Even though they have high accuracy, these models are static and cannot adapt to changes in the environment. To maintain the accuracy of the prediction models, we have adopted Online Learning techniques that update the models on data from subjective viewer feedback. Overall this method provides accurate and adaptive QoE prediction models that can become an indispensible component of a QoE-aware management service. Keywords Quality of Experience, QoE, Machine Learning, Online Learning, QoE-aware management, Usercentric management 1. Introduction Advances in the telecommunication systems open opportunities for multimedia services of higher quality, which were previously demanding excessive network resources. As these services are becoming more common many service providers are facing the problems of their efficient management. Streaming multimedia services such as IP TV or Video conferencing have high resource demands and stringent requirements. Efficient management of multimedia services depends on understanding the value they bring to the viewers, which in turn depends on the service perceived quality. The perception of quality from a particular multimedia service is closely related to many factors such as the image fidelity, image resolution, type of device, content, audio fidelity. Traditional approaches to network service management solely focus on the transport and encoding quality such as the Quality of Service parameters and neglect many additional factors. Thus, many multimedia services are underor over-provisioned. The encoding parameters are not adapted to the presentation device or the type of content, and none take into account the user’s expectations. As this data-centric service management is not aware of the customer’s perceived quality, it cannot be as efficient with the system’s resources as user-centric management approach. To improve the service management, a shift to a user-centric or user-aware multimedia service management is necessary (Agboma & Liotta, 2008). Menkovski, V., Exarchakos, G., Liotta, A., & Sánchez, A. C. (2010). Quality of Experience Models for Multimedia Streaming. International Journal of Mobile Computing and Multimedia Communications (IJMCMC), 2(4), 1-20. doi:10.4018/jmcmc.2010100101 To execute user centric management a model for the perceived quality or Quality of Experience (QoE) is necessary. QoE is a subjective metric that quantifies the perceived quality of a service by the viewers. As such, QoE needs to correlate numerous parameters that affect the perceived quality such as the encoding, transport, content, type of terminal, as well as the user’s expectations (Agboma & Liotta, 2007). The QoE management approach aims at maximizing the perceived quality of the viewers while minimizing the impact on the system’s resources. Since QoE is a subjective metric the most accurate estimation methods are execution of subjective studies and calculation of the Mean Opinion Score (MOS) values from user feedback. However, subjective studies involve a complex selection procedure of an appropriate and statistically viable testing group and of the exact test conditions. Organizing such tests is a cumbersome and expensive effort and is not feasible for live streaming content. The method proposed here builds QoE prediction models based on data from initial limited subjective studies. Building the prediction models is done with Machine Learning (ML) algorithms that yield highly accurate QoE models which require low processing power for execution. To sum up, the proposed QoE prediction models aim at maintaining high perceived quality at the user end with a low resource cost on the delivery network. The product of these models is the QoE estimation to be used for fine-tuning of service parameters. This can be an on-going process as viewers’ expectations or conditions change over time with introduction of new content and viewing devices. Environment alterations cause their accuracy to drop, rendering new subjective tests necessary. Online Learning techniques, adopted for maintaining the prediction accuracy, can help width building adaptive models updated with subjective feedback from the users. Our results show that the models adapt quickly to changes (with a small number of feedback responses) and reach high accuracy closely comparable to the static models. Overall this method provides a means for estimating the QoE of multimedia content using prediction models and the ability to keep those models accurate in dynamic environments. 2. QoE Measurement Methods Perception of quality in multimedia streaming is getting more attention with the proliferation of high quality multimedia technologies. The reason for this is twofold; on the one hand, user’s expectation of quality has increased and, on the other, managing the larger demand on the network resources has become a more pressing matter. There are many aspects and dimensions of perceived quality but the focus of this paper, regarding the multimedia services, is the QoE as a comprehensive quality metric. A number of definitions for QoE exist, but in general most descriptions agree that QoE is what the end-user experiences while using the service. Even though there is more or less an agreement on how to define QoE, there is no single view on how to measure it. Many different approaches have been proposed, some of which fundamentally different. Though there is an agreement that QoE is a subjective metric (Takahashi, Hands, & Barriac, 2008), due to the drawbacks of subjective analysis (S Winkler, 2007), most efforts focus on objective methodology in measuring QoE. The typical approach for measuring QoE is estimating the quantity of errors that appear in the presented content. These errors are due to compression artifacts, transportation impairments and combinations of both. There is a wide range of models for QoE estimation; the focus of these models is spread over different points in the path of the content from the creation to the presentation. Due to this diversification some standardization bodies have made attempts to standardize these models. In (Takahashi et al., 2008) the International Telecommunication Union (ITU) presents a classification of the different objective quality assessment models. The classification is made into the following types: media layer, Menkovski, V., Exarchakos, G., Liotta, A., & Sánchez, A. C. (2010). Quality of Experience Models for Multimedia Streaming. International Journal of Mobile Computing and Multimedia Communications (IJMCMC), 2(4), 1-20. doi:10.4018/jmcmc.2010100101 parametric, bit-stream, and hybrid models. This classification is based on the model’s point of interest. The media layer models focus on the media signal and uses knowledge of the Human Visual System (HVS) to predict the subjective quality of video. The parametric models predict the quality by looking at protocol information and network statistics, acquired with non-intrusive probes. The bit-stream models derive the quality via analyzing content characteristics collected from the coded bit-stream information. In a survey of different video quality methods (Stefan Winkler, 2009) the author concludes that there are many different methods and algorithms for QoE estimation but there is a lack of a standard for comparing their capabilities. Models that only look at the fidelity of the audio and video estimate the QoE based on the signal distortion. This methodology remains oblivious to the content of the media as well as to the workings of the HVS. A typical example is a pixel to pixel comparison such as the Peak Signal to Noise Ratio (PSNR) and Mean Squared Error (MSE) methods. The drawback of these methods is that they compare the signals without any understanding of the HVS or how that content is perceived (S Winkler, 2007). There are many cases on which PSNR delivers far from accurate results. One of the simplest examples is shifting the image into any direction by one pixel. This will lower the value of the PSNR significantly but the perceived quality by a human will be basically intact. The media-layer models focus on the content of the stream itself. They implement objective perceptual video quality measurement by modeling the workings of the HVS (Stefan Winkler, 2005). These models are computationally expensive because they need to execute in-depth analysis of the media content. Even though media-layer models show better performance than the data accuracy focused methods, no accurate objective method has been developed yet that takes into account all the physiological and psychological aspects of quality perception (Wang, Bovik, & Lu, 2002). Modeling the effects that the transport has on the delivered quality would mean looking at the Quality of Service (QoS) parameters. This approach is not very efficient and yields weaker results (Siller & Woods, 2003). The authors of (Siller & Woods, 2003) propose looking at the problem in three layers. The bottom layer is the network layer. The network layer produces the QoS parameters or more precisely the Network QoS (NQoS) parameters. The layer above presents the application layer which is concentrated on parameters such as resolution, frame rate, color, and codec type. These parameters are referred to as Application QoS (AQoS). The third or the top layer is the perception layer which is driven by the human perception of the multimedia content and is concentrated on spatial and temporal perception and acoustic bandpass (Siller & Woods, 2003). The QoE, which is measured on the top layer is a function of both AQoS and NQoS (1). QoE = f(AQoS, NQoS)..................................................(1) In the proposed framework (Siller & Woods, 2003), the authors discuss that arbitrating all of the QoS parameters together is significantly more effective in maximizing the QoE than looking at each of them individually. Due to the subjectiveness of QoE, the most accurate way to measure it is by executing a subjective test. Subjective studies are of significant importance because they can accurately convey the satisfaction of the viewers with the service. This is why subjective tests are commonly used for comparing the capabilities of different QoE estimation methods. Subjective testing usually entails execution of tests in a tightly controlled environment with a carefully selected group of subjects, which represent the population that is using the service. Guidelines for the execution of different subjective studies are provided by the ITU (ITU-T, 1999). Menkovski, V., Exarchakos, G., Liotta, A., & Sánchez, A. C. (2010). Quality of Experience Models for Multimedia Streaming. International Journal of Mobile Computing and Multimedia Communications (IJMCMC), 2(4), 1-20. doi:10.4018/jmcmc.2010100101 The drawbacks of subjective studies are obvious from their description. They require significant effort and resources to be put into their design and execution. (Agboma & Liotta, 2007) presents a method that only relies on initial limited subjective tests. From the results of these tests, statistical models are build that can predict the QoE on unseen cases. This approach is suitable for minimizing the need for cumbersome subjective studies while providing for estimation based on the user’s subjective feedback. The weakness of this approach is the limited statistical method for building the prediction models. The following sections present the design of prediction models using ML techniques that will achieve significantly accurate estimations of QoE. Expanding this method with ML Online Learning techniques based on continual viewer feedback, the prediction models become more flexible and adaptable to changes of the environment. Figure 1. QoE Prediction Method 3. QoE Prediction Models The method of estimating QoE using prediction models consists of gathering subjective data, feeding it into the prediction model induction algorithm and using that model to predict the QoE (Figure 1). So, the first phase is to implement the data acquisition. In (Agboma & Liotta, 2007) and (Agboma & Liotta, 2008) the executed subjective tests are based on the Method of Limits (Fechner, Boring, Adler, & Howes, 1966). This method is used to detect the thresholds for quality by changing a single stimulus in successive, discrete steps. A series terminates when the intensity of the stimulus becomes detectable. When, according to the user, the quality decreases bellow the satisfactory level, the test series stops. The aim is to determine the user thresholds of acceptability and the QoS parameters taking into account content and terminal type. Table 1 depicts the different video samples that the users were subjected to. Each segment is a row in the tables and has corresponding QoS parameters given in the column values. Table 1. Test-bed combinations for 3G Mobile Phone descending series (174 x 144 image size), for PDA descending series (320 x 240 image size), for Laptop descending series (640 x 480 image size) Segment Time (seconds) Video bitrate (kbit/s) Audio bitrate (kbit/s) Framerate M ob ile 1 1-20 384 12.2 25 2 21-40 303 12.2 25 3 41-60 243 12.2 20 4 61-80 194 12.2 15 5 81-100 128 12.2 12.5 6 101-120 96 12.2 10 7 121-140 64 12.2 6 8 141-160 32 12.2 6 PD A 1 1-20 448 32 25 2 21-40 349 32 25 3 41-60 285 32 20 4 61-80 224 32 15 5 81-100 128 32 10 6 101-120 96 32 10 7 121-140 64 32 6 8 141-160 32 32 6 La pt op 1 1-20 448 32 25 2 21-40 349 32 25 3 41-60 285 32 20 4 61-80 224 32 15 5 81-100 128 32 10 6 101-120 96 32 10 7 121-140 64 32 6 8 141-160 32 32 6 Figure 2. QoE Levels for different content types on all terminals The results of the subjective tests given in Figure 2 show the dependency of the QoE on the type of terminal as well as the content. For instance, user’s expectations for content such as football are Menkovski, V., Exarchakos, G., Liotta, A., & Sánchez, A. C. (2010). Quality of Experience Models for Multimedia Streaming. International Journal of Mobile Computing and Multimedia Communications (IJMCMC), 2(4), 1-20. doi:10.4018/jmcmc.2010100101 different on the different terminals. On the other hand, the type of content with low dynamics such as a news broadcast has high perceived quality even with low bandwidth consumption on all devices. In addition, the audio quality is significantly more important than video quality in most content types, but particularly in news (Agboma & Liotta, 2008). It is of great importance to capture these correlations in order to derive the dependencies of the QoS and other parameters to the QoE. In (Agboma & Liotta, 2008) and (Agboma & Liotta, 2007), the correlations are captured with statistical models as discussed in the next section. 3.1 Statistical models for QoE Statistical analysis of the subjective tests is a technique that can be used to build models from the subjective data. In (Agboma & Liotta, 2008) the Discriminant Analysis method (Klecka, 1980) was used to build the prediction models. This method builds linear functions for each class or label with which each data point can be associated. The class of the function that returns the maximum value is associated with the class of the data point. In this analysis the data was divided in subsets for each terminal and then in smaller subsets for each content type. The discriminant functions are built based in two input parameters Video Bitrate and Video Framerate. In Figure 3 two classification discriminant functions are given. One for the news content and the second for the action movie, both of which are for the mobile terminal. Given the bitrate and the framerate of the news broadcast video, these functions, h(!""#$%!&'#) !"#$%&"'()'$,!" and h(!"#$$%&'#()%) !"#$%&"'()'$,!" , can calculate the acceptance degree of a video. If, for example, h(!""#$%!&'#) is larger than h(!"#$$%&'#()%), the News broadcast is predicted to have acceptable QoE and vice versa. That work generated prediction models for all of the listed content types for these three terminals. The accuracies of those models, validated with the leave-out-one method for each terminal averaged, are (Agboma, 2009) • Mobile phones: 76.9% • PDA: 86.6% • Laptop: 83.9% The following section carries on with the motivation and implementation of a different statistical analysis method for building QoE prediction models based on Machine Learning techniques which show superior accuracy. 3.2 ML QoE Prediction Models Machine Learning is mainly focused on developing algorithms for induction of models based on training data. These models are commonly used for pattern recognition and decision support. As is the case here, these techniques build models for estimation of QoE based on subjective feedback data. The algorithms used belong to the group of supervised learning algorithms. The training data is classified by a human or acquired experimentally. In our case the two classes to which the data News h(acceptable) = ‐5.699 + (-­‐0.080×VideoBitrate) + (1.613×FR) h(unacceptable) = ‐4.223 + (-­‐0.104×VideoBitrate) + (1.613xFR)

DOI: 10.4018/jmcmc.2010100101

Extracted Key Phrases

3 Figures and Tables

Citations per Year

Citation Velocity: 6

Averaging 6 citations per year over the last 3 years.

Learn more about how we calculate this metric in our FAQ.

Cite this paper

@article{Menkovski2010QualityOE, title={Quality of Experience Models for Multimedia Streaming}, author={Vlado Menkovski and Georgios Exarchakos and Antonio Liotta and Antonio Cuadra S{\'a}nchez}, journal={IJMCMC}, year={2010}, volume={2}, pages={1-20} }