Dynamic Scheduling of Real-Time Mixture-of-Experts Systems on Limited Resources

A Mixture-of-Experts (MoE) system generates an output in each operating cycle by combining results of multiple models (the “experts”). The contribution of any given expert to a final solution depends on a parameter called responsibility, which can vary from cycle to cycle. When resources are insufficient to run all experts, two problems arise: 1) how much utilization is to be allocated to experts and 2) how can a schedule be created based on these allocations. Problem (1) can be formulated as a succession of optimization problems, each of which calculates experts’ allocations in a cycle. Explicit mappings from responsibilities to allocation weights are needed to solve each of these problems in every cycle using a technique called “task compression (TC).” We refer to this baseline approach as TT-TC. Two other proposed heuristics
${\ssr TT}\hbox{-}{\ssr TC}^\ast$
and TT-Top reduce TC’s execution time to
${\ssr O}({\mbi{N}})$
for
${\mbi{N}}$
experts. To address (2), the proposed EPOC scheduler converts the heuristics’ allocations into schedules that satisfy capacity, execution, and learning constraints across cycles. Simulations demonstrate that our approaches enable real-time computation and significantly decrease the average percentage error of limited-resource outputs (i.e., 0.2%–40% and 0.3%–0.5% when scheduled with
${\ssr TT}\hbox{-}{\ssr TC}^\ast$
and TT-Top, respectively, versus 0.2%–97% when using TT-TC).