A Systematic Literature Review on the Characteristics and Effectiveness of Web Application Vulnerability Scanners

Web applications have been a significant target for successful security breaches in the last few years. They are currently secured, as a primary method, by searching for their vulnerabilities with specialized tools referred to as Web Application Vulnerability Scanners (WVS’s). Although, these dynamic approaches of testing have some advantages, there is still a scarcity of studies that explore their features and detection capabilities in a systematic way. This article reports findings from a Systematic Literature Review (SLR) to look into the characteristics and effectiveness of the most frequently used WVS’s. A total of 90 research papers were carefully evaluated. Thirty (30) WVS’s were collected and reported, with only 12 having at least one quantitative assessment of effectiveness. These 12 WVS’s were evaluated by 15 original evaluation studies. We found that these evaluations tested mostly only two of the Open Web Application Security Project (OWASP) Top Ten vulnerability types: SQL injection (SQLi) (13/15) and Cross-Site Scripting (XSS) (8/15). Additionally, only one work evaluated six of the OWASP Top Ten vulnerability types and for only one scanner. We also found that the reported detection rates were highly dissimilar between these 15 evaluations. Based on these surprising results we suggest avenues for future directions.