Rating Duration Analysis for Subjective Quality Assessment of 360° Videos

Due to large advances in immersive media systems offering increased resolutions, frame rates, and dynamic ranges, applications such as watching 360° videos on head-mounted displays (HMDs) have become more popular. To assist the development of immersive media applications, subjective tests of 360° videos on HMDs are needed to obtain a ground truth on the quality as perceived by the end-users. The absolute category rating (ACR) method, standardized for subjective quality assessment of conventional videos, has also been used for assessing the subjective quality of 360° videos. Recently, the modified ACR (M-ACR) method has been proposed aiming at subjective quality assessment of 360° videos. Several subjective test campaigns have been conducted to assess the ACR and M-ACR method for evaluating 360° videos covering a wide range of resolutions, bit rates, and quantization parameters. The opinion scores given by the participants in these subjective tests are accumulated to mean option scores (MOSs) and confidence intervals. However, to the best of our knowledge, a detailed statistical analysis and comparison of the rating durations associated with casting opinion scores for 360° test videos in the ACR and M-ACR method has not been reported in the literature. In this paper, we therefore provide a rating duration analysis for subjective quality assessment of 360° videos for the ACR and M-ACR method. The results support experimental designs for 360° video quality assessment and include the following findings: 1) The mean rating duration is shorter for the ACR method, 2) The difference of the standard deviation of rating durations versus MOS between the ACR and M-ACR method is insignificant, 3) For a given mean rating duration, however, the standard deviation is smaller for the M-ACR method indicating that the quality rating task is easier to execute than with ACR. 4) The period to be allocated in subjective tests for rating 360° videos should exceed 5 s but can be kept below or set equal to 10 s.