Columbus: Enabling Scalable Scientific Workflows for Fast Evolving Spatio-Temporal Sensor Data

As spatiotemporal sensor data collections are becoming more available, the ability to generate insights from them is turning increasingly complex for scientists in this domain. Organizing such voluminous data and keeping track of analysis results get tedious from day to day. In this paper, we propose a system comprising a distributed workflow execution engine, a storage framework with flexible spatiotemporal data dispersion and indexing scheme for fast data search and retrieval, and a scheduling scheme that aims at minimizing data movement across machines while simultaneously ensuring fair and efficient allocation of resources among multiple users. Our empirical evaluations demonstrate the feasibility of our approach and its ability to scale.