Dynamic data transformation for low latency querying in big data systems

Big data storage technologies inherently entail high latency characteristics, preventing users from performing efficient ad-hoc querying and interactive visualization on large and distributed datasets. Most of the existing approaches addressing this issue thrive on de-normalization of the static data schema and creation of application specific (i.e. hard-coded) materialized views, which certainly reduce data access latency but at the expense of flexibility. In this regard, this paper proposes an approach that relies on an iterative process of data transformation intended to generate read-optimized data schemas. The transformation process is able to automatically identify optimization opportunities (e.g. materialized views, missing indexes), by analyzing the original data schema and the record of queries issued by users and client applications against the data set. An experimental evaluation of the proposed approach evidences a significant reduction in the query latency, ranging from 81.60% to 99.99%.