Performance Analysis of NVMe SSD-Based All-flash Array Systems

In this paper, we analyze and optimize I/O latency of a petabyte scale, high performance all-flash array (AFA) system based on NVMe SSDs. A flash-based SSD itself shows relatively low and consistent latency but, in AFA systems where several tens or hundreds of SSDs are combined in a single host machine, applications often see higher and more diverged I/O latency compared with a standalone SSD. To figure out a main source of such high I/O fluctuations, we analyze end-to-end I/O latency characteristics of a real-world AFA system. We find out that suboptimal kernel policies, parameters, and configurations result in serious degradation of I/O response times, causing very long tail latency. Based on our observations, we manually reconfigure several kernel parameters and revise storage firmware to achieve consistent I/O latency. Our experimental results show that, with the finely tuned kernel for AFA systems, the mean and standard deviation of the maximum latency can be reduced by x8 and x400, respectively. The findings in this work provide useful wisdom in designing system software and operating systems - CPU schedulers need to be revised to take into account the priority of IO-bound jobs, CPU isolation, and CPU-SSD affinity, and moreover, storage housekeeping protocols like SMART should be improved to avoid long tail latency.