Predictive Disk Provisioning for Adjustable Cloud Storage Solutions

Cloud service providers such as AWS and Azure have recently begun to offer storage solutions that allow disk performance to be adjusted “on-the-fly”. Such offerings allow the user to make use of short-term predictions of storage requirements. For example, instead of provisioning a single storage solution that is never under-provisioned, but frequently over-provisioned, one can configure the storage system to support higher performance during peak times; and cheaper, lower performance during periods with less demand. This paper explores the possibility of using a prediction system that utilizes past storage demands to predict the storage requirements over the next hour. We sought a single predictor that could perform well for all types of demand. The predictors were developed using approximately 200 years of storage performance requirements collected from high-performance storage systems in hundreds of companies. We have found that over-provisioning can be greatly reduced, but only at the expense of under-provisioning with a non-zero probability. However, the probability of being under-provisioned can be as low as 0.01%, which is similar to the target service level of cloud vendors. In addition, we have developed novel methods to search for effective predictors that perform well both on average and for rare events.