Towards End-to-End QoS and Cost-Aware Resource Scaling in Cloud-Based IoT Data Processing Pipelines

Ensuring cost-effective end-to-end QoS in a multilayer, multi-service, IoT data processing pipeline is a non-trivial challenge. The uncertainties surrounding the 3Vs of streaming data - variety, velocity and volume - impose dynamic QoS-driven resource requirements on each component (or service) of the pipeline and make adaptive resource management a complex task. Our overall research objective is to develop appropriate resource scaling strategies that dynamically adjust the resources allocated to each component in the pipeline so as to ensure end-to-end QoS fulfillment while optimizing the associated costs. To this end, in this paper, we present our work in progress on a model for end-to-end QoS and cost-aware resource allocation for IoT data processing pipelines. We base our model on the well-established unbounded knapsack problem, which offers a simple yet powerful abstraction of constraint-based decision-making. We intend to develop resource scaling strategies on top of this model that can exploit resource and contract heterogeneity to achieve cost-optimal end-to-end QoS-aware resource allocations.