Can source code auditing software identify common vulnerabilities and be used to evaluate software security?

Software vulnerabilities are a growing problem (c.f. MITRE's CVE, http://eve.mitre.org). Moreover, many of the mistakes leading to vulnerabilities are repeated often. Source code auditing tools could be a great help in identifying common mistakes, or in evaluating the security of software. We investigated the effectiveness of the auditing tools we could access, using the following criteria: number of false positives, false negatives by comparison to known vulnerabilities, and time required to validate the warnings related to vulnerabilities. Some of the known vulnerabilities could not be found by any code auditor, because they were fairly unusual or involved knowledge not contained or codified in the source code. The coding problems that could be identified consisted of string format vulnerabilities, buffer overflows, race conditions, memory leaks, and symlink attacks. However, we found it extremely time-consuming to validate warnings related to the latter four types, because the number of false positives was very high, and because it was not easily apparent if they were real vulnerabilities. These required that the code be audited locally, by people familiar with the code, and carefully inspected to see if the values could be manipulated in such a way as to produce malicious effects. However, the string format vulnerabilities were much easier to recognize. In small and medium scale projects, the open source program Pscan was useful in finding a mix of coding style issues that could potentially enable string format vulnerabilities, as well as actual vulnerabilities. The limitations of Pscan were more obvious in large scale projects like OpenBSD, as more false positives occurred. Clearly, auditing source code for all vulnerabilities remains a time-consuming process, even with the help of the current tools, and more research is needed in identifying and avoiding other common mistakes.