Evaluation of static analysis tools for software security

Security has been always treated as an add-on feature in the software development lifecycle, and addressed by security professionals using firewalls, proxies, intrusion prevention systems, antivirus and platform security. Software is at the root of all common computer security problems, and hence hackers don't create security holes, but rather exploit them. Security holes in software applications are the result of bad design and implementation of software systems and applications. To address this problem, several initiatives for integrating security in the software development lifecycle have been proposed, along with tools to support a security-centric software development lifecycle. This paper introduces a framework for evaluating security static analysis tools such as source code analyzers, and offers evaluation of non-commercial static analysis tools such as Yasca, CAT.NET, and FindBugs. In order to evaluate the effectiveness of such tools, common software weaknesses are defined based on CWE/SANS Top 25, OWASP Top Ten and NIST source code weaknesses. The evaluation methodology is based on the NIST Software Assurance Metrics And Tool Evaluation (SAMATE). Results show that security static analysis tools are, to some extent, effective in detecting security holes in source code; source code analyzers are able to detect more weaknesses than bytecode and binary code scanners; and while tools can assist the development team in security code review activities, they are not enough to uncover all common weaknesses in software. The new test cases developed for this research have been contributed to the NIST Software Assurance Reference Dataset (samate.nist.gov/SARD).