Optimization of the Processing of Data Streams on Roughly Characterized Distributed Resources

The AS4DR (Adaptive Scheduling for Distributed Resources) scheduling method presented in this paper aims at maximizing throughput, when processing several data streams by divisible load applications on star-shaped distributed memory platforms, with available speeds for communicating and computing which may be poorly estimated, or varying over time. The total workload is supposed to be unknown. According to the computation cost model, AS4DR can either maximize throughput, or CPU utilization by avoiding data-starvation of the computing units. An experimental assessment of the adaptation of the workload distribution to the variation of the communicating and computing speeds has been performed that shows that the use of AS4DR can significantly improve the throughput. This paper also experimentally assesses a resource selection method to set up star-shaped clusters of distributed resources, so as to process efficiently a set of data streams with AS4DR.