Formal validation of the security properties of AMT's three-way handshake

AMT (Automatic IP Multicast without explicit Tunnels) is a specification that has been developed by the Internet Engineering Task Force to address the lack of multicast communication among isolated multicast-enabled sites or hosts, attached to a network with no local multicast support. AMT is designed to provide a mechanism for a migration path to a fully multicast-enabled backbone in the future. As part of a larger project using AMT to extend the reach of multicast sessions, we have performed formal validation of the three-way handshake process between an AMT gateway and its coupled AMT relay by modeling it using the AVISPA tools (Automated Validation of Internet Security Protocols and Applications). We have identified two security problems where an intruder can impersonate an AMT Relay or an AMT Gateway. Furthermore, an intruder can make use of this impersonation to disconnect valid sessions of other legitimate participants.