Extending CometCloud to Process Dynamic Data Streams on Heterogeneous Infrastructures

Coordination of multiple concurrent data stream processing, carried out through a distributed Cloud infrastructure, is described. The coordination (control) is carried out through the use of a Reference net (a particular type of Petri net) based interpreter, implemented alongside the Comet Cloud system. One of the benefits of this approach is that the model can also be executed directly to support the coordination action. The proposed approach supports the simultaneous processing of data streams and enables dynamic scale-up of heterogeneous computational resources on demand, while meeting the particular quality of service requirements (throughput) for each data stream. We assume that the processing to be applied to each data stream is known a priori. The workflow interpreter monitors the arrival rate and throughput of each data stream, as a consequence of carrying out the execution using Comet Cloud. We demonstrate the use of the control strategy using two key actions - allocating and deal locating resources dynamically based on the number of tasks waiting to be executed (using a predefined threshold). However, a variety of other control actions can also be supported and are described in this work. Evaluation is carried out using a distributed Comet Cloud deployment - where the allocation of new resources can be based on a number of different criteria, such as: (i) differences between sites, i.e. Based on the types of resources supported (e.g. GPU vs. CPU only, FPGAs, etc), (ii) cost of execution, (iii) failure rate and likely resilience, etc.