Balancing Human and Machine Performance When Analyzing Image Cover

In computer vision, it is often unclear how humans should annotate data in a way that optimizes both human effort and machine performance. This is particularly true in image cover applications, in which the goal is to determine what percentage of the image is covered by a certain class of objects (e.g. trees or grass). In this paper, we present the first study comparing the efficacy of fundamentally different annotation styles when applied to image cover problems. Our results on two real-world marine science problems indicate that the most accurate human annotations do not necessarily result in the best machine learning performance, and that by carefully selecting the format of the annotations one can achieve good performance even with limited human effort.