Navigator: A Decentralized Scheduler for Latency-Sensitive AI Workflows

We consider AI inference and generative query processing in distributed systems: an increasingly popular edgecomputing use case. In such systems, coscheduling of GPU memory management and task placement represents a promising opportunity. We propose Navigator, a novel framework that unifies these functions to reduce job latency while using resources efficiently, placing tasks where data dependencies will be satisfied collocating tasks from the same job (when this will not overload the host or its GPU), and efficiently managing GPU memory Comparison with other state of the art schedulers shows a significant reduction in completion times while requiring the same amount or even fewer resources. In one case, just half the servers were needed for processing the same workload.