Larry Suto, an application security consultant, publish a sequel to his 2007 best seller research about web application scanners. In the first round Larry managed to ignite quite a controversy and drew a lot of criticism from the loosing vendors. The reason is simple: Larry found out that the scanners do not perform as well as advertised. While the delta is under debate, the basic premise that the scanners do not provide a bullet proof solution for securing web applications was easily demonstrated.

In this round Larry expands the coverage and correct many of the shortcomings of the previous paper. In a quite ironic twist he tests the scanners against the site their vendors provide for tests, which one would assume is perfectly covered by the scanner. To his dismay the scanners still miss a lot.

Larry's report is a rare pearl in an analyst and media community that has complicated relations with vendors and in many cases insufficient technical level and resources. It is so rare that someone actually tests in detail how well a security solution work.

Why is this of interest to the WAF community? The naive answer would be that scanners and WAFs are alternatives. While they do not perform the same function, they compete for the same budget and are offered as alternatives by PCI DSS.  If scanners are not as good as expected, WAF might be the right solution after all. This is especially important as WAFs are usually under more fire than scanners as it is much simpler to find a fault in a WAF - just find the right evasion vector. For a scanner a full analysis as done by Suto is required.

However the paper has other more far reaching conclusions on the state of security products in general and therefore WAFs:

• No single security solution is sufficient. Only combining multiple defense mechanism would provide adequate security, which still does not imply 100%
• Security products do differ in the security functionality they provide. Many times customers select security products according to every other feature but security assuming that the security aspect of the product are performed adequately by all. However Suto's paper shows that this may not be the case.
• The lack of scrutiny of the security features drive security vendors to neglect security and focus on other areas such as GUI, reporting or manageability. This is shown in its extreme by the inability of some scanners to find existing vulnerabilities in sites provides for testing by the vendor itself.