Performance comparisons of subjective quality assessment methods for mobile video

There are many subjective assessment methods, depending on the purpose of the video quality evaluation, provided by ITU-T and ITU-R recommendations. We compared eight subjective assessment methods, Double-Stimulus Continuous Quality-Scale (DSCQS), Double-Stimulus Impairment Scale (DSIS), ACR 5-grade scale (ACR5), ACR5 with Hidden Reference (ACR5-HR), ACR 11-grade scale (ACR11), ACR11-HR, Subjective Assessment of Multimedia Video Quality (SAMVIQ), and SAMVIQ-HR, to clarify their applicability to a variety of video quality assessment. To do this, we evaluated the total assessment time and difficult/ease in evaluation for the participants, as well as the characteristics of different rating scales and their statistical reliability. As a result, we clarified that the correlation coefficients and rank correlation coefficients of the mean opinion scores between pairs of the eight methods were high. As for statistical reliability of each rating method, DSIS, ACR5, and SAMVIQ outperformed the other methods. Moreover, participant questionnaire results showed that ACR5 was a suitable method from the viewpoint of total assessment time and ease of evaluation.