Optimizing Performance Using GPU Cache Data Residency Based on Application’s Access Patterns

Memory management is a significant aspect of executing applications on GPUs even in the cloud environment. With the advancements in GPU architecture, issues such as data reuse, cache line eviction and data residency are to be considered when optimal performance for concurrently running applications. Frequency of data access from global memory has significant impact on the performance of the application with increased latencies when accesses result in cache misses. Through static profiling, we identify the access patterns to the global memory and investigate the relationship between frequent access patterns and data residency in the cache. From our investigations, we observed that each application frequently accesses a data region in memory though the range of addresses accessed differ. We evaluated our estimated set-aside area for LSTM and CSR applications. Executions using our proposed estimations shows a speed-up in the performance LSTM (1.004x) while CSR experienced a slow-down (0.998x) when both were co-executed with their respective estimated set-aside areas.