Research Opportunities for the Big Data Era of Software Engineering

Big Data Analysis is becoming a widespread practice on many software development projects, and statisticians and data analysts are working alongside developers, testers and program managers. Because data science is still an emerging discipline in software projects, there are many opportunities where software engineering researchers can help improve practice. In terms of productivity, data scientists need support for exploratory analysis of large datasets, relief from clerical tasks like data cleaning, and easier paths for live deployment of new analyses. In terms of correctness, data scientists need help in preserving data meaning and provenance, and non-experts need help avoiding analysis errors. In terms of communication and coordination, teams need more approachable ways to discuss uncertainty and risk, and support for data-driven decision making needs to become available to all roles. This position paper describes these open problems and points to ongoing research beginning to tackle them.