Invited Talk: The Role of Empiricism in Improving the Reliability of Future Software

This talk first bemoan the general absence of empiricism in the evolution of software system building and then go on to show the results of some experiments in attempting to understand how defects appear in software, what factors affect their appearance and their relationship to testing generally. It challenge a few cherished beliefs on the way and demonstrate in no particular order at least the following: 1) the equilibrium state of a software system appears to conserve defect; 2) there is strong evidence in quasi-equilibrated systems for xlogx growth in defects where x is a measure of the lines of code; 3) component sizes in OO and non-OO software systems appear to be scale-free, (this is intimately related to the first two bullet points); 4) software measurements, (also known rather inaccurately as metrics) are effectively useless in determining the defect behaviour of a software system; 5) most such measurements, (including the ubiquitous cyclomatic complexity) are almost as highly correlated with lines of code as the relationship between temperature in degrees Fahrenheit and degrees Centigrade measured with a slightly noisy thermometer. In other words, lines of code are just about as good as anything else when estimating defects; 6) 'gotos considered irrelevant'. The goto statement has no obvious relationship with defects even when studied over very long periods. It probably never did; 7) checklists in code inspections appear to make no significant difference to the efficiency of the inspection; and 8) when you find a defect, there is an increasing probability of finding another in the same component. This strategy is effective up to a surprisingly large number of defects in youthful systems but not at all in elderly systems.