Security Testing: Turning Practice into Theory

This position paper proposes a research agenda for the field of security testing. It gives a critical account of the state of the art as seen by a practitioner and identifies questions that research failed to answer so far, or failed to answer in such a way that it would have had an impact in the real world. Three categories of research problems are proposed: theory of vulnerabilities, theory of security testing, and tools and techniques. 1. About this Paper The science of security testing is still in its infancy. This paper proposes a research agenda for this field. It does so from a very specific perspective: that of a tester who, being aware of the lack of a scientific basis of his work, has to and wants to assess the security level of software systems on the basis of testing. What such a tester needs is not research papers but useful tools that optimize the work that is already being done in various labs around the world. The key underlying assumption of this paper is therefore that research should take an approach similar to what a usability engineer would do when designing a tool: first understand the task, then design solutions and tools. Hence the title, turning practice into theory. This paper contains no original research whatsoever. Rather, it is a position paper and conveys the author's opinion on the subject. The author has a background in applied research and practical security testing, which may explain some of the views expressed here. Primarily, the present paper collects problems the author encountered during several years of testing and evaluating systems for their security. Secondarily it presents a number of observations how security testing is approached today, none of which should be taken for more than anecdotal evidence, though. The remainder of this paper is organized as follows: Section 2 outlines the author's conception of security