Capacity Management for Streaming Applications over Cloud Infrastructures with Micro Billing Models

Recent advances in sensor technologies and instrumentation have led to an extraordinary growth of data sources and streaming applications. A wide variety of devices, from smart phones to dedicated sensors, have the capability of collecting and streaming data at unprecedented rates. Typical applications include smart cities & built environments for instance, where sensor-based infrastructures continue to increase in scale and variety. Analysis of stream data involves: (i) execution of a number of operations on a time/sample window - e.g. min./max./avg., filtering, etc, (ii) a need to combine a number of such operations together, (iii) event-driven execution of operations, generally over short time durations, (iv) operation correlations across multiple data streams. The use of such operations does not fit well in the per-hour or per-minute cloud billing models currently available from cloud providers - with some notable exceptions (e.g. Amazon AWS). In this paper we discuss how micro-billing and sub-second resource allocation can be used in the context of streaming applications and how micro-billing models bring challenges to capacity management on cloud infrastructures.