Agile big data analytics: AnalyticsOps for data science

Big data analytic (BDA) systems leverage data distribution and parallel processing across a cluster of resources. This introduces a number of new challenges specifically for analytics. The analytics portion of the complete lifecycle has typically followed a waterfall process - completing one step before beginning the next. While efforts have been made to map different types of analytics to an agile methodology, the steps are often described as breaking activities into smaller tasks while the overall process is still consistent with step-by-step waterfall. BDA changes a number of the activities in the analytics lifecycle, as well as their ordering. The goal of agile analytics - to reach a point of optimality between generating value from data and the time spent getting there. This paper discusses the implications of an agile process for BDA in cleansing, transformation, and analytics.