Query-Adaptive Online Partitioning of Associated Data for Efficient Retrieval

Data partitioning is a crucial component of any distributed storage system that wants to scale. For retrieval efficiency, data frequently requested together in the same query should be placed on the same server as much as possible. Although intuitive, this is not easy to be implemented if constrained by load balancing, computationally, it is an NP hard problem. Existing research has offered approximate solutions optimized for a given workload of queries, in which the order as to when each query is received is not considered. This paper initiates a new study on online partitioning algorithms that are sequentially optimized for a query sequence. In the new problem, the queries arrive in a stream manner, unknown, and given the option to revise the partition after each query, the objective is to minimize the total query processing cost and data migration cost. We formulate this problem formally, investigate several online heuristics, and evaluate them using simulation.