Zero-shot Binarization Lightweight Model for a Data-Free Edge Network

Recently, edge networks are developed to be integrated with computation and storage in a harmonious architecture to perform intelligence tasks efficiently. However, there are two major challenges in implementing smart computation in edge networks. (1) An intelligent model should be extremely lightweight, because of limited computing power at edge. (2) Data-driven model training requires the protection of security and privacy at edge. To march towards meeting those two challenges, we propose a zero-shot binary neural network (ZS-BNN), which is an ultimate compact model without data-driven training in this paper. With ZS-BNN, a slim BNN client performs data-free adversarial distillation (DFAD) training by introducing a generator and a full-precision teacher to servers. Moreover, considering the distribution deviation, two mechanisms are proposed for further optimization, where the BN constraint term is investigated for distribution rectification in generators, and a distribution alignment for student BNNs is elaborated. Based on those designs, ZS-BNNs have normal data distributions via improvement operations. As a result, binary convolution model is implemented unprecedentedly in a data-free environment and can be considered the most available solution for edge network intelligence. The experimental results show that ZS-BNN is superior to state-of-the-art binarization methods in zero-shot scenarios. The resource codes are validated at https://github.com/sjmp525/IA/tree/ZS-BNN.