Malicious Relay Node Detection with Unsupervised Learning in Amplify-Forward Cooperative Networks

This paper presents malicious relay node detection in a cooperative network using unsupervised learning based on the received signal samples over the source to destination (S-D) link at the destination node. We consider the situations in which possible maliciousness of the relay is the regenerative, injection or garbling type attacks over the source signal according to attack modeling in the communication. The proposed approach here for such an attack detection problem is to apply unsupervised machine learning using one-class classifier (OCC) algorithms. Among the algorithms compared, One-Class Support Vector Machines (OSVM) with kernel radial basis function (RBF) has the largest accuracy performance in detecting malicious node attacks with certain types and also detect trustable relay by using specific features of the symbol constellation of the received signal. Results show that we can achieve detection accuracy about 99% with SVM-RBF and k-NN learning algorithms for garbling type relay attacks. The results also encourage that OCC algorithms considered in this study with different feature selections could be effective in detecting other types of relay attacks.