AC-DNN: An Adaptive Compact DNNs Architecture for Collaborative Learning Among Heterogeneous Smart Devices

With the rapid development of the Internet of Things (IoT), a massive number of smart devices are deployed in industry and critical infrastructures. Nowadays, IoT smart devices have drawn increasing attention for collaborative learning tasks, e.g., persistent monitoring and online recognition. In this article, we present an adaptive compact deep neural network (DNN) approach (termed as AC-DNN) to tackle the challenging problem of unreliable transmission for collaborative learning tasks among heterogeneous smart devices. We introduce a cross-platform model weight encoding, decoding, and dispatching architecture to accommodate to differential smart devices and improve the reliability of intermediate model transmission via encapsulating binary model weights into self-contained transactions. To decrease encoding and decoding overhead, we design a quantile-based histogram sketch to compress the intermediate model. We conduct extensive evaluations to test our AC-DNN framework and deploy the AC-DNN on federated learning testbed FedAvg. We evaluate our approach functionality using different DNN architectures, such as convolutional neural network and ResNet and compare their effectiveness within the different network structures. The experiments reveal that our approach can improve the reliability of collaborative learning tasks among smart devices. Meanwhile, we can achieve nearly 70% weight compression compared to the original model size with minimal loss of accuracy. Our approach facilitates the deployment of a DNN-like network among discrete mobile smart devices for deep and persistent learning tasks.