Tales of the Tail: Past and Future

Tail latency has been the defining performance metric that interactive cloud services have had to meet from the beginning of the cloud as a computing paradigm. While there has been a wide spectrum of techniques, in hardware and software, that have improved tail latency for these applications, recent trends across the cloud system stack require revisiting them. Over the past few years, cloud hardware has become increasingly heterogeneous, and cloud software has been dominated by event-driven modular programming frameworks, as well as the proliferation of AI. To guarantee tail latency in this new landscape, several system advances are required. In this paper, we first review what tail latency means for cloud services, the key innovations that improved it in the past, the trends that require revisiting them, as well as the innovations that will be required for tail latency constraints to be met in the next generation of warehouse-scale computers.