Advancing the Automation Capability of Verifying Security Protocols

Current formal approaches have been successfully used to find design flaws in many security protocols. However, it is still challenging to automatically analyze protocols due to their large or infinite state spaces. In this paper, we propose SmartVerif, a novel and general framework that pushes the limit of automation capability of Tamarin, a state-of-the-art protocol verifier. The primary technical contribution is the dynamic strategy inside SmartVerif, which can be used to smartly search proof trees. Different from the existing static strategies, our dynamic strategy can automatically optimize itself according to the security protocols without any human intervention. We implement the strategy by modifying Tamarin and introducing a reinforcement learning algorithm to avoid non-terminating paths in the proof tree. Besides, to improve SmartVerif, we add multiple extracted information for training the reinforcement learning network and design a submodule of Non-termination Estimation to collect training data precisely and rapidly. Experimental results show that SmartVerif can automatically verify all security protocols studied in this paper. The case study validates the efficiency of our dynamic strategy. The experimental results also demonstrate the effectiveness of our extracted information, and the accuracy of the submodule of Non-termination Estimation.