Pipelined memory hierarchies: scalable organizations and application performance

The time to perform a random access to main memory has been increasing for decades relative to processor speed and is currently of the order of a few hundred cycles. To alleviate this problem, one resorts to memory organizations that are hierarchical to exploit locality of the computation, and pipelinable to exploit parallelism. The goal of the study is to begin a systematic exploration of the performance advantages of such memories, achieving scalability even when the underlying principles are pushed to the limit permitted by physical laws. First, we propose memory organizations with the ability to accept requests at a constant rate without significantly affecting the latency of individual requests, which is within a constant factor of the minimum value achievable under fundamental physical constraints. Second, we discuss how the pipeline capability can be effectively exploited by memory management techniques in order to reduce execution time for applications. We conclude by outlining the issues that require further work in order to pursue systematically the potential of pipelined hierarchical memories.