Framework for evaluating Capture the Flag (CTF) security competitions

A large number of ethical hacking competitions are organized worldwide as Capture The Flag (CTF) events. But there does not exist a framework to evaluate and rank CTFs that will guide participants as to which CTF's to participate. In a CTF event, the participants are required to either solve a set of challenges to gain points or they are required to defend their system by eliminating the vulnerabilities while attacking other's system vulnerabilities. We are proposing a framework that would evaluate and rank CTFs according to factors like similarity of the tasks to the common critical vulnerabilities, solvability of tasks, periodicity, training given prior to CTF, geographical reach, problem solving skills etc. In the next step these factors are systematically assigned weights using Analytic Hierarchy Process. As part of frame work creation and validation, ten CTFs have been analysed. Our analysis indicates that: All CTFs fall in to one of the three categories (jeopardy, attack-defence and mixed); CTFs often adopt popular software vulnerabilities and threats as tasks to be solved; Only few CTFs give formal training prior to the event; Complexity of the tasks to be solved varies from CTF to CTF. Five CTFs were ranked using the newly developed framework.