Analyzing the Variability of Subjective Image Quality Ratings for Different Distortions

When it comes to evaluating the quality of images, individual observers show different opinions depending on the type of distortion affecting the quality of the image. While in the opinion of one observer a distortion could have a dramatic influence on the quality of the image, another observer could see the same distortion as not having an important effect on the quality of the same image. Using a subjective experiment, we aim to identify the distortions which show the largest variability among observers. For this, 22 observers evaluated the quality of 10 reference images and the 630 test images created from them (21 distortions at three levels). Our results show that the highest variability in subjective scores is linked to distortions like saturation, contrast, sharpness, quantization, some types of added noise, and radial lens distortion.