Transparent In-memory Cache Management in Apache Spark based on Post-Mortem Analysis

This paper proposes an extension to Apache Spark that provides automated and efficient in-memory cache management based on post-mortem dependency graph analysis. This extension allows programmers to focus on algorithmic issues without code modification for caching decision. We realized this extension with two techniques: (1) a selection algorithm for intermediate data to be cached and (2) a cache replacement algorithm looking ahead to the whole execution. For avoiding all recalculations in case of no cache replacement, the selection algorithm implicitly activates the necessary cache directives. When cache replacement is unavoidable, the cache replacement algorithm prevents frequent cache replacement that came from excessive cache directives. Experimental results demonstrate that machine learning applications on the extended runtime achieved competitive or higher performance up to 1.3 times compared to manually optimized programs, except for very simple cases. We expect that the proposed method is useful to reduce the cache management burden on programmers without regard to the amount of processing data.