EPIC Collab: Supporting Asynchronous Collaboration in Big Data Analysis Systems

The rise of big data has led to the creation of large datasets that require teams to collaborate to analyze data effectively. Unfortunately, the software systems that collect and analyze large datasets are not often designed to support this kind of collaboration. Accordingly, our work investigates issues related to supporting collaboration in big data analysis systems. We use the domain of crisis informatics and the software infrastructure of Project EPIC as a case study to gain insight into the features that analysts need to effectively perform analysis at scale. This paper focuses on supporting asynchronous collaboration among analysts who work in small distributed teams on big data software systems. It describes the challenges faced by researchers who work collaboratively to analyze large crisis datasets (consisting, typically, of Twitter data). It then describes the work performed to redesign an existing big data analysis environment to substantially improve its support for collaboration. The impact of this research lies in its ability to improve the work of similar teams performing large-scale data analysis. While our work is based on insights gleaned from crisis informatics, we believe that our design, results, and lessons learned are broadly applicable to other application domains.