A novel mutual authentication protocol for H(e)NB and mobile devices using S8 S-box

In wireless communication, femtocell is a feasible option to enhance the signal strength. Authentication of the legal user and legitimate femtocell is a serious issue. Many protocols have been proposed for mutual authentication but each has its own pros and cones. In this paper we first investigate the weakness of a previously presented mutual authentication protocol, Rapid Development and Authentication Protocol (RDAP) under Man In The Middle (MITM), Denial of Service (DoS), sybil and masquerade attacks. Then, a novel authentication protocol is presented to remove the weakness against these attacks. The proposed algorithm is interoperable and use S8 S-box for AES to reduce the authentication time.