Human-Survey Interaction (HSI): A Study on Integrity of Human Data Collectors in a Mass-Scale Hajj Pilgrimage Survey

Mass gatherings (such as Hajj/Umrah), owing to their immensity, often present a variety of difficulties to the attendees. To have a comprehensive understanding of the difficulties as well as their potential remedies, surveying a good number of attendees is unavoidable and this can be facilitated through engaging data collectors. A crucial part here is identifying the integrity of the data collectors, which is yet to be explored in the literature to the best of our knowledge. To address this gap, in this study, we first perform a mass-scale data collection over Hajj/Umrah pilgrims through online (n = 236) and in-person (n = 752) surveys, where we cover a substantial part (n = 712) through paid data collectors (n = 53). We critically investigate data collection activities of the data collectors through focused group discussions involving expert reviewers. We explore two computing techniques to unveil the integrity of the data collectors from two different perspectives. Our study finds out influential (religious and socio-geographical) aspects that impact the data collection process. Besides, we find that the collaborative participation of expert reviewers is obligatory to scrutinize the data collectors' integrity. Additionally, our explored computing techniques, namely conflict analysis and learning-based analysis, can identify up to 74% and 99% of the unreliable data collectors respectively. We observe that, although these computing-based filtering can indicate the integrity up to a certain level, human-in-the-loop is unavoidable for concluding on the integrity. To the best of our knowledge, this is the first study of its kind (i.e., integrity analysis of the data collectors) in the literature.