Will academic security research provide the answer?

Industry research such as Larry Suto’s is often superficial and at times driven by external motivations. So where should we derive our research data from? While one answer may be to forgo with research data all together, another option that comes to mind is academic research. An academic paper comparing vulnerability discovery techniques that I encountered is a good test case for that. In the paper Austin et al compare four different methods for finding vulnerabilities in web applications: static analysis, automated pen-testing, manual pen-testing and systematic manual pen-testing. The conclusion is that no two methods provide much overlap and therefore one should use all, and if human efficiency is taken into account prefer systematic manual pen-testing and automated pen-testing as they give most bang for the buck.

When compared with Suto’s paper I analyzed last week, Austin et al immediately presents many methodological advantages typical to academic work and rigorously controlled by the academic paper mandatory structure, among them:

• Comprehensive study of prior work (including two Suto papers, an issue requiring a separate discussion).
• More reasonable test design, including the choice of software used (market leaders vs. Suto’s niche player that does not support the methodology for half of the tested environments) and software tested (real world applications vs. scanner vendors test web sites).
• Scrupulous reporting of methodology and results. While this is an area in which Suto attempts to emulate scientific research, he fails to do so in critical elements as discussed in my dissect.
• Acknowledgement of limitations, additional observations that were not considered in the analysis and source of funding.

In addition to these specific differences, the impression made by the paper also suggests it is more exploratory and does not have specific dogma in mind to prove regardless of evidence.

While all these make academic papers A much better source of information security research, there is also a downside. One clear caveat of academic papers is that they are much harder to read. Leaving that aside, Austin et al exhibits many of the limitations of the Suto’s paper. The culprit is that the research belongs to a family called “case studies” in which the authors select a single or very limited set of environments and generalized based on the results. In this research one of testing software or each type, two tested systems and a single person (for scenarios in which human efficiency was measured) were used. More over the variance in the results between the different methods may suggest the invalidity of the research rather than the paper’s conclusion that the methods are mutually exclusive: for example, it may point that the selected tested systems are not typical or that the vulnerability assessment was done inadequately. Austin et al acknowledge all those limitations in the paper, but continue to draw conclusions anyway.

So are we to know? Things may not be that bad. Keep in mind that I repeated here the same fault as I used a single case study for my analysis also. Unfortunately reading any research paper including academic ones requires considerable care. I urge you to read  Ben Goldacre's Bad Science (the book and the blog). While Ben, a physician by training and a science journalist by trade, talks about medical and nutritional research, many of his observations would serve you well in dissecting information security research as well.