Suto strikes again, or getting the desired results regardless of data

Larry Suto, who brought scanners wars 1 and 2 has published a WAF effectiveness research. As usual, Larry’s work is fun to dissect. While Larry’s research is not worse than your average analyst’s work, he does try to base his conclusion on more concrete and pseudo-scientific research making it much more vulnerable to scrutiny.

The first thing that raised my eyebrow was that IPS did as well as WAFs given scanner training. One would argue that this implies you really don’t need a WAF. As usual, the devil is in the details. Reading through the document I found that Larry decided to leave out testing WAF evasion. Sitting in Iftach Amit’s presentation at Source Barcelona learning about attacks involving obfuscated text modulated as voice and sent to a Google voice mailbox, this immediately struck me as lackluster. Larry is testing IPS and WAFS against the last millennia’s challenges. Evasion is so basic to hacking today making a benchmark that ignores it useless. More specifically, I think that evasion is a key difference between IPS and WAFs, and the conclusion that the two solutions are comparable is entirely based on this research omission. As a matter of fact, Larry identifies this issue by concluding that the examples of evasion he is aware of are all around pattern matching, the fundamental IPS technology, but does not draw the inevitable conclusion.

Another major claim the study makes is that training by a scanner makes perfect. This fails both research methodology and interpretation examination. While the measured value of scanner training for IPS was impressive, the value for WAFs is arguable: for one WAF the value was marginal, for another it was intermediate and for 3 scanners training was not tested. The only reason the average value seems OK is the last WAF, ModSecurity, for which the benefit of training is high. On the statistical side, the large variance and small set make the conclusion invalid. On the technical side, I would argue that ModSecurity behaves much more as an IPS, as this research proves. However those interpretation issues pale in front of the major methodology flow: the same technology was used to train and test!

While this analysis undermines two out of the 3 conclusions of the research, the 3^rd just cannot be analyzed as the research does not provide any supporting information that can be examined. While it is reasonable to assume that expert setup of a WAF considerably enhance the security it provides, concluding based on this work that 3.5 hours is the amount of time to invest is flaky at best. Does it depend on the protected application size and complexity? It’s rate of change? What was the variance for the tested systems? Does it make the results statistically significant?

To sum up, Larry cannot base his conclusions on the research he did. This does not imply that the conclusions are wrong, just not proved by this study. A final important note is that Larry is not alone, and I am yet to see any so called security research that is not pseudo research in the security world. Maybe it is just not possible.