FairCIM: Fair Interference Mitigation by DNN Switching for Latency-Sensitive Inference Jobs

It is a common practice to co-locate several jobs, such as DNN inference ones, on a single machine to improve resource utilization, energy efficiency, and overall performance. However, individual job performance might degrade when these jobs compete for shared resources. This degradation can in particular affect latency-sensitive jobs bound by Service Level Agreements (SLA), and result in SLA violations. To tackle this issue and mitigate the impact of interference on latency, we design a system called FairCIM. For certain applications such as image classification, various DNNs with varying accuracy and computational complexity are proposed. Upon detection of SLA violations, FairCIM dynamically switches between those DNNs to maintain the latency in the presence of co-location interference. To ensure fairness, FairCIM selects the jobs for DNN switching based on their resource utilization. We conduct experiments using diverse DNNs, datasets, and hardware platforms to evaluate the FairCIM effectiveness. Compared to another approach that uses a resource exchange mechanism, FairCIM’s flexibility achieved by DNN switching substantially reduces SLA violations and improves the effective accuracy by up to 244%.