WAVSEP Updates, FAQ and the 2015 Benchmark Roadmap

A couple of updates on the WAVSEP 2015 benchmark:

The 2015 benchmark is already ongoing, and I started testing scanners against a newer unpublished version of WAVSEP which will be published at the end of the benchmark.

I'll be focusing on the usual commercial and actively maintained open source contenders, but may include additional vulnerability scanner engines that match my criteria or join the comparison in one of the methods listed in SecToolMarket.

WAVSEP New Homepage

As of August 2015, WAVSEP has been official migrated to github, and the various installation instructions have been migrated to the relevant github wavsep wiki (installation / features).

The source code, builds and wiki will be maintained in github, but I'll be releasing builds to wavsep sourceforge repository as well.

Just to clarify - both repositories currently contain the latest public version of WAVSEP.

About the Upcoming Benchmark

The benchmark will cover all the previously covered aspects, as well as 2-3 additional attack vectors, and 2-3 new measurement concepts. Its the biggest one so far, but hopefully, I'll find smarter methods of assessing the products to speed up the process.

As mentioned before, to make the results useful earlier, I'll be publishing some of the results during the testing to SecToolMarket, and tweet when there's updates to the various engines, instead of waiting to the end of the benchmark.

Vulnerability Scanner Feature Mapping to RvR

The plan is to eventually associate the various features assessed in WAVSEP with a new project called RvR (relative vulnerability rating), currently hosted in the following address, aimed to define identical classifications of features for comparing security products.

The RvR list still includes 288 (!) attack vectors with videos, links, etc, but there's already 60+ additional attacks pending to be added, contributed by volunteers from around the globe.

Trying to Release an Initial WAFEP Benchmark

WAFEP (Web Application Firewall Evaluation Project), WAVSEP's evil WAF testing brother, is almost ready for initial release, with thousands of proven WAF bypass payloads ready.
However, I'm trying to release an initial benchmark with the framework, covering 2-5 WAF engines to make my point.

Its tricky to stuff these projects in the same timeframe, and WAVSEP is my current priority, but we'll see how it works - WAFEP is designed to take a lot less testing time.

In any event, I'll tweet about additional updates and whenever I update the results.

Cheers

RvR, WAFEP and WAVSEP results update

Most of my time these days is spent on creating a dynamic interface for updating benchmark results, and on two major projects aimed at enhancing the WAVSEP evaluations and adding additional comparison content, in addition to accuracy, crawling and automation.


The first project, RvR (Relative Vulnerability Rating), is a project I already mentioned in the past which merges vulnerabilities from well known vulnerability classifications (WASC, CWE, CAPEC, OWASP, Blogs, Conferences, etc) into a list customized specifically for product feature evaluations.

The list, originally planned to include 233 attack vectors, already includes 284 (!!!) different attack vectors with unique classifications, links, repository mapping and videos,
A web site containing the content was published last week, and although all the content is very much usable, I'm still delaying the publication until I get some vendor feedback (expect an official publication soon).

The purpose of the project is not only to evaluate features of dynamic vulnerability scanners (DAST), but also to cover source code analysis tools (SAST), interactive application testing tools (IAST), and in contrast to the past - various software protection products, including application-level IDS/IPS mechanisms and web application firewalls (WAF).

Which leads me to the second project -

WAFEP - The Web Application Firewall Evaluation Project

WAFEP is an upcoming project aimed to serve a WAVSEP-like role for various application-level protection products.

Unlike WAVSEP, WAFEP is planned on being completely automated in terms of payload execution AND result calculation, and would enable the evaluation of web application firewalls in relatively short timeframes.

The "accuracy" aspect is implemented as attack vector specific payloads meant to simulate context-specific exploits that an IDS/IPS/WAF should identify and/or prevent, false positive scenarios that should not be identified, and in the future, evasion techniques that may circumvent the detection process.

The project already includes thousands of payloads imitating flavors of +-10 high-impact attack vectors, some of which were already published in an early alpha version uploaded to the project source forge repository last week.

The published alpha version is just a technology POC, and does not include most of the vector payloads or content, but in the upcoming weeks I'll make an effort to finish up some sections in the platform and release a v1.0 public version.
I'll also publish updated versions with relevant payloads in the meantime, at least until I reach the 1.0 goal.

http://sourceforge.net/projects/wafep/


WAVSEP Results Update

Finally, from time to time, I still try to squeeze in additional WAVSEP product assessments for additional vendors, the latest of which is Tinfoil Security, alongside certain version upgrades,

As always, the full list is found in SecToolMarket, and the following image summarizes the updates:

If all goes well, in the near future, the list will be updated with the results of a couple of more.

I didn't update the results of any of the open source products, and will try to find the time to do so in the near future, at least for some of the projects - a task that should be much easier once the dynamic interface is finally online.