Confidence in the Qualified Crowd: A Platform for Sourcing EEG Annotations

The use of new signal processing and machine learning algorithms to find digital biomarkers in electroencephalogram (EEG) signals requires annotations for EEG signals at scale and in an on-going manner. In this paper, we provide an overview of the computational infrastructure to support this with an emphasis on how we efficiently process EEG signal data from multiple sources and manage all of the corresponding information. If each annotator is better than 50% accurate, increasing the number of annotators for annotation tasks will result in higher and higher reliability of the annotation. Our platform enables new approaches to quantifying the accuracy of annotations based on the annotators' accuracy, expertise, and relevant experience.