Time-Critical Cooperative Delivery with Unknown Demands

This paper studies the Time-Critical Cooperative Delivery with Unknown Demands (TCDUD) problem, developed from the well-studied Capacitated Vehicle Routing Problem with Stochastic Demands (CVRPSD), that corresponds to emergency situations requiring 1) time-critical delivery; 2) unknown demands; and 3) cooperative delivery. We formulate the problem as a multi-agent sequential decision problem with a Collaborative Semi-Markov Decision Process (CSMDP) model in which timecritical delivery is urged by the reward function that takes both the amount and the time of fulfilled demands into account. Unlike traditional CVRPSD, we do not include a priori information about a customer's demand in problem definition, and require vehicles to visit a customer before its demand is revealed. A Transformer-based multi-agent reinforcement learning approach, namely MAODN, is devised to learn online policies that direct the vehicles to visit the customers, and perform timely delivery in a cooperative manner. MAODN utilizes a demand updater to accommodate online updates about customers' demands from the vehicles during delivery. Vehicles then make cooperative decisions via individual policy networks, leveraging the fleet state provided by the state aggregator. Experiment results suggest that our approach outperforms the baseline by at least 27.8% and demonstrates better robustness.