Dynamic Caching for Multi-Job and Real-Time IoT Data Analytics

With the advent of the Internet of Things (IoT), the volume of real-time data has surged, necessitating efficient data analytics frameworks like Spark to handle the deluge. Traditional caching strategies, which rely on either historical data or anticipated access patterns, often overlook the complexities of parallel computations in IoT environments, resulting in suboptimal performance. Our paper introduces a dynamic caching approach tailored for multi-job IoT data analytics that leverages both historical and future access patterns, accounting for parallel processing. By proposing the concepts of critical path, hot data and representative RDD, we simplify the caching problem and convert it to a knapsack problem. Our proposed algorithm, DCSR, employs dynamic programming to optimize caching decisions. Experimental results demonstrate that DCSR significantly outperforms existing methods, reducing job completion time by at least 25.94%.