Towards High Performance Processing of Streaming Data in Large Data Centers

Smart devices, mobile robots, ubiquitous sensors, and other connected devices in the Internet of Things (IoT) increasingly require real-time computations beyond their hardware limits to process the events they capture. Leveraging cloud infrastructures for these computational demands is a pattern adopted in the IoT community as one solution, which has led to a class of Dynamic Data Driven Applications (DDDA). These applications offload computations to the cloud through Distributed Stream Processing Frameworks (DSPF) such as Apache Storm. While DSPFs are efficient in computations, current implementations barely meet the strict low latency requirements of large scale DDDAs due to inefficient inter-process communication. This research implements efficient highly scalable communication algorithms and presents a comprehensive study of performance, taking into account the nature of these applications and characteristics of the cloud runtime environments. It further reduces communication costs within a node using an efficient shared memory approach. These algorithms are applicable in general to existing DSPFs and the results show significant improvements in latency over the default implementation in Apache Storm.