A gold standard for assessing the coverage of static analyzers

Static analyzers can be used at compile time to automatically discover locations in programs that are vulnerable to malicious exploits. It is difficult, however, to compare the results of one analyzer to another, or to assess the coverage of any one analyzer against the actual number of vulnerabilities present in a program, because there is no analyzer-independent metric of vulnerabilities for software. We are currently using our CodeHawk abstract interpretation technology to produce such a formal metric for 6 C programs from NIST's SATE competition. A single mathematical definition of undefined memory access according to the C semantics covers 27 of the MITRE Common Weakness Enumeration categories, and allows us to identify a set of program locations where proof obligations guaranteeing memory-safety cannot be discharged. These locations comprise a canonical enumeration of the weaknesses against which to measure static analyzers, including true positives (correctly identified locations), false positives (identified locations that can be proved safe), and false negatives (unsafe locations not identified).