A case study in detecting software security vulnerabilities using constraint optimization

In this paper we present a case study in static analysis, with a focus on static methods for detecting buffer overflow vulnerabilities in software. We describe in detail a tool called Mjolnir that we have developed which improves upon existing static analysis techniques for detecting buffer overflow. The architecture and process flow of this tool are presented We discuss some common static analysis obstacles in terms of where they were encountered in developing this tool and the steps that were taken to overcome them. A prototype of the tool has been implemented and used for detecting buffer overflow vulnerabilities in C programs and experimental results are presented that demonstrate the effectiveness of the tool.