Spark Framework for Real-Time Analytic of Multiple Heterogeneous Data Streams

Real-time streaming applications with multiple heterogeneous data streams have become increasingly popular especially in IoT applications; however, many issues still exist, especially in deploying and maintaining these large amounts of data streams. Using Spark Structured Streaming, this paper introduces a Spark Streaming framework for multiple heterogeneous data streams which allows the deployment of multiple heterogeneous data stream processing in a single Spark application; reducing deployment difficulty, coding redundancy, monitoring difficulties, and solving the problem of inefficient job queueing in multi-stream applications.