An Empirical Study of QoE Estimation for Video Streaming Services Using Crowdsourcing

The challenge of evaluating and modeling Quality of Experience (QoE) for content streaming video services is discussed in this paper. Based on the QoE evaluation methodology proposed by the EU-funded TRIANGLE project, a novel no-reference QoE model is presented to predict the user Mean Opinion Score (MOS) for videos streamed services such as YouTube. The study has been performed in a testbed that can control radio and network parameters, enabling real-life scenarios for mobile applications and devices in a controllable and repeatable manner. This work takes advantage of Amazon Mturk as a crowdsourcing platform to collect user opinions at scale, efficiently and quickly harvesting opinion scores from a much more diverse population than it would be possible from a local human panel. The performance of the proposed no-reference model is compared with PEVQ-S, a full-reference standardized algorithm used in the industry, showing better performance but with a much simpler approach. The performance has also been evaluated in SQoE-III, a publicly available adaptive video streaming database, to ensure the robustness of the model.