Data-driven Synchronization Protocols for Data-parallel Neural Learning over Streaming Data

We introduce EVENFLOW, a novel toolkit of synchronization protocols for data-parallel training of neural nets using the Parameter Server (PS) paradigm. EVENFLOW achieves both timely and accurate global model updates in streaming settings. Instead of leaving stragglers out of the global model to avoid delays (asynchronous protocol) or using laggy synchronizations of all learners (synchronous protocol), EVENFLOW establishes data-driven mechanisms that allow the PS paradigm to decide when a synchronization is necessary, i.e., the global model may have changed beyond an allowed tolerance value. EVENFLOW models this problem as a distributed, thresholded function monitoring task and decomposes it to local filters monitored independently by each learner. When a learner finds its local filter violated, only then a synchronization is triggered. Our experiments show that EVENFLOW combines the virtues of both the vanilla (synchronous, asynchronous) protocols. EVENFLOW offers the rapid training times of asynchronous, with mostly equal or even improved accuracy compared to synchronous.