Multilayer Probabilistic Knowledge Transfer for Learning Image Representations

Probabilistic Knowledge Transfer (PKT) aims to transfer the knowledge encoded in the representations extracted from a layer of a large and complex neural network (teacher) into a smaller and faster one (student). However, PKT only transfers the knowledge between two layers of the networks, ignoring the potentially useful information encoded by the previous ones, reducing in this way the efficiency of PKT and the performance of the student model. In this paper, we propose a novel efficient multilayer PKT method that is capable of transferring the knowledge between the student and teacher networks by employing the representations extracted from multiple layers. The ability of the proposed multilayer PKT method to improve the knowledge transfer and increase the performance of the student model over other state-of-the-art methods is demonstrated using two image datasets.