Performance Analysis of NVMe SSD-Based All-flash Array Systems

In this paper, we analyze and optimize I/O latency of a petabyte scale, high performance all-flash array (AFA) system based on NVMe SSDs. A flash-based SSD itself shows relatively low and consistent latency but, in AFA systems where several tens or hundreds of SSDs are combined in a single host machine, applications often see higher and more diverged I/O latency compared with a standalone SSD. To figure out a main source of such high I/O fluctuations, we analyze end-to-end I/O latency characteristics of a real-world AFA system. We find out that suboptimal kernel policies, parameters, and configurations result in serious degradation of I/O response times, causing very long tail latency. Based on our observations, we manually reconfigure several kernel parameters and revise storage firmware to achieve consistent I/O latency. Our experimental results show that, with the finely tuned kernel for AFA systems, the mean and standard deviation of the maximum latency can be reduced by x8 and x400, respectively. The findings in this work provide useful wisdom in designing system software and operating systems - CPU schedulers need to be revised to take into account the priority of IO-bound jobs, CPU isolation, and CPU-SSD affinity, and moreover, storage housekeeping protocols like SMART should be improved to avoid long tail latency.

I. Introduction

NVMe SSD-based all-flash array (AFA) systems recently receive serious attention from industry because their low latency and high throughput can aid various applications such as high-performance computing and real-time big data analysis [1]. An AFA also offers plenty of other benefits over disk storage - it consumes less power, does not require as much cooling as HDDs, and takes up less floor space. Thanks to all those benefits, an AFA is emerging as fast and economic storage solutions, rapidly replacing existing HDD-based storage clusters in data centers.