DAST vs. SAST vs. IAST - Modern SSLDC Guide - Part I

Disclaimer This article uses a relative ratio for the various charts, to emphasize the ups and downs of various technologies to the reader. It also reflects the current situation to date (which may change as technologies mature), and relies on generalization’s and estimations on capabilities of technologies, and so, must be read in the proper context.

Using a variety of vulnerability detection solutions have become widespread in software development projects, with the aim of detecting crucial vulnerabilities as early as possible.

The vast collection of technologies and tool-sets available to address the issue can dazzle even an expert, raising questions such as -

The engimous "Which technology is the best ?"

The intriguing "Do DAST/SAST/IAST complement or supersede each other ?"

And the inevitable "How to prioritize their usage and acquisition ?"

With the upcoming publication of the WAVSEP 2017 benchmark close at hand, I wanted to take the opportunity to provide my take on the role of DAST tools within the  context of prominent technologies and trends in the field.

Where do they fit in? What do they excel in (compared to alternatives)? When should we use them?

With the introduction and maturity of new vulnerability-detection technologies (IAST/DAST/SAST/HYBRID/OSS), and the expected streamline of (understandably) conflicting vendor claims, users may find it hard to discern which technologies may fit their needs, how to PRIORITISE their acquisition/integration, and when’s the right time to engage each solution category.

In the following article, I will be covering a few of the key aspects in the integration of these toolsets into an SSDLC (secure software development life cycle) environment – the OVERALL EFFORT and the IDEAL TIMING of each solution category, and the benefit of SUPPORTED TECHNOLOGIES and CODE COVERAGE they provide under different cirucmstances.

In addition to being exposed to a wide variaty of tools-of-the-trade, the article can also help the reader answer some basic questions when evaluating any one of these tools.

For those of us who somehow managed to escape the terms currently in use – this article covers the following technologies:

1) DAST – Dynamic Application Security Testing – Generic and Known Web Application Vulnerability Scanners that analyze a live application instance for security vulnerabilities. To further clarify – this is the category of tools that was covered in all the previous WAVSEP benchmarks.

• This article specifically focuses on DAST solutions which are actively maintained and/or SSDLC adapted, with the ability to verify potential vulnerabiliteis through some sort of Exploitation/Verification process (referred to as EV for the purposes of the article), either external or in built into the detection algorithm, as opposed to "fuzzer" like tools based primarly on algorithms that rely on identifying specific keywords in the response.

2) SAST – Static Application Security Testing – Generic & Known Application Vulnerability Code-Level Scanners that analyze source code and application configuration files for security vulnerabilities.

3) IAST – Interactive Application Security Testing – Generic and Known Application Vulnerability Debug/Memory Level Analysis Solutions that attempt to identify vulnerabilities on live application instances while also analyzing code structures in the memory and tracking the input flow throughout the application sections. This category is further divided into the following subcategories:

• Passive IAST – IAST solutions that rely on traffic already being generated to identify potentially vulnerable sections, WITHOUT performing additional attack/exploit verifications (e.g. sending input with all the necessary exploitation characters, etc).
• Active IAST – IAST solutions that verify potential vulnerability sinks/sources through the use of requests that verify the actual exploitability of the potential vulnerability (again, by issuing requests that contain input with all the necessary exploitation characters, or through similar means).

4) OSS - Open Source Security – the SAST equivalent of the mythological CGI-scanner – these solutions that were, for the purposes of this article, integrated into the category of SAST, due to similarity of the chart positioning and role, although these solutions operate in an entirely different manner, and focus only on the identification of “known” vulnerabilities in 3^rd party libraries. 

*) The various aspects of hybrid analysis tools are NOT covered in the various article sections and charts, and the same goes for network vulnerability scanners with application level features but without SSDLC adaptation, or cloud security solutions without SSDLC integrations.

So Which Solution Category Is Most Important in SSDLC?

Technology Support vs. Code/Application Coverage

The most obvious differentiation between the various scanning solution categories is the amount of supported technologies - as in - which development languages are supported. 

After comparing the various official and unoffical supported technologies proclaimed at the various vendor data-sheets, we'll quickly get to the following conclusion:

IAST solutions typically support only a handful of development technologies (2017 stats), SAST solutions can support a myrid of modern and legacy programming languages, and DAST solutions are rarely affected by the development technology -

Click to Enlarge

#	Supported Technologies
DAST	Any application with WEB/REST/WebService back-end. Some exotic back-end listeners may be supported as well (web-sockets, DWR, AMF, etc). The support also depends on compatibility with input delivery vectors, as well as compatible crawling OR session recording features.
SAST	Java, ASP.Net, C#.Net, VB.Net, PHP, Noje.js, Html/JS, SQL, Ruby, Pyhon, C, C++, JSP, ASP3, VB6,  VBScript, Groovy, Scala, Perl, Apex, VisualForce, Android/iOS/WinMobile, Objective C, Swift, PhoneGap, Flex ActionScript, COBOL, ABAP, Coldfusion CFML
IAST	Java, .Net, PHP (few vendors), Node.js (few vendors), Ruby (few vendors), Python (experimental)

* data gathered from the proclaimed/documented features of the vast majority of existing products (May 2017)

The charts and tables signify the result of high end DAST/SAST/IAST in the industry, and obviously, some SAST and IAST solutions may support a much lower subset of technologies than the listed scope. Comparing the support for scanning non-web application variants (custom non-HTTP-based protocols) will drastically affect the chart as well.

It is also important to mention that DAST/Active IAST solutions in automated modes also need to be able to "crawl" the technology, or at the very least support the creation of recorded "sessions" of a manual crawling process, and also support sending attack payloads through the "input delivery vectors" used by the application (e.g. query-string / body / JSON/ XML / AMF / etc).

Although SAST / Passive IAST solutions also need to support "tracking" the input delivery vectors, they could, theoretically, identify hazardous code patterns without tracking the entire input-output flow, with the price of potential false positives being reported.

The difference in technology support in IAST solutions is partially related to the fact that IAST implementations are relatively new compared to DAST or SAST implementations, but also to the amount of effort required to "integrate" the IAST engine to each new technology, and furthermore, to maintain the implementation with the release of newer versions of the same supported technologies (adaptations may be required for major java JVM versions, newer .net framework versions, etc).

So, although this technology-support "GAP" may be smaller over time, the effort that will be required to maintain technology support will grow at a bigger pace, at least when compared to the pace of DAST and SAST technology compliance.

It is however, worth mentioning that most IAST vendors focus on widely used technologies that would cover as much ground as possible (Java / .Net / PHP / Node.js), and thus the actual importance of this "gap" will greatly vary for some organizations, and may even be insignificant for some.

And what of coverage ?

As it appears, being able to "support" a technology, does not necessarily allow the testing tool to automatically cover the larger portion of it's scope, which in turn, may dramatically affect the results. 

The type of technologies evaluated, the method of evaluation, the code deployment format, and even the "legal" ownership of the source code libraries may affect the actual sections being covered by the tool.

The coverage criteria will be easier to understand in the form of a table, rather than a chart:

Coverage	DAST	SAST	Passive-IAST	Active-IAST
Out-Of-The-Box Wide Coverage Min Effort	? In Unauthenticated/ Form/Basic/NTLM	✔ Most Scenarios	? In Tested/Used Instances	X Depending on Implementation
End-To-End Coverage Scan/Correlate Issues in All Client/FE/BE Layers	✔ In Client Triggered Sequences	X Depending on Implementation	X Depending on Implementation	? Depending on Implementation
3rd Party Code Closed Source Libraries/Entry-Points	? For “Visible” Methods	X No DE-compilation	✔ Depending on Implementation	✔ Depending on Implementation
Dead/Blocked Code Non-Web Executable	X Depending on Implementation	✔ Most Scenarios	X Depending on Implementation	X Depending on Implementation

Conclusion:
DAST and SAST tools *typically* support more technologies, and as far as coverage is concerned -

• DAST excels in end-to-end coverage (As in scanning the FULL CYCLE of front-end to backend) AND "visible" 3rd-party coverage, but may require manual configuration for complex applications, or at the very least, an effective crawling mechanism that supports the front-end GUI technology.
• SAST excels in out-of-the-box coverage, but may lack in 3rd party software coverage (assuming it does not perform de-compilation of 3rd-party libraries), and may requires manual syncing to "identify" associated end-to-end layers. That being said, early in development, it's probably the most likely method of getting early feedback on potential vulnerabilities.
• IAST will typically be positioned somewhere between the two in the various coverage categories - it will require agent distribution to support end-to-end detection (if it is supported at all), but will require less effort to achieve a wide coverage of application entry points (particularly in the case of Passive-IAST), and might have the advantage of potentially providing an in-depth coverage for CLOSED 3rd-party code/libraries.

Integration Effort vs. False Positives Effort

Throughout the development process, in both the early and later stages, the amount of effort invested in detecting vulnerabilities can, knowingly or not, play a key role in the success of the SSDLC process.

For every vulnerability detection solution and for every scenario, resources are required to integrate the chosen solutions, maintain the integration (not as easy as it sounds), and go over the results to filter high-impact and relevant issues.

Since for all the phases there’s a limited amount of human and IT resources, overly complex integrations can DELAY (or sometimes even PREVENT) the detection of security issues to a point that the benefit of detecting them early won’t apply, while complex and tedious result analysis processes can easily cause the developers to ignore identified critical issues due the sheer number of irrelevant results.

The overall effort of using each tool, is not always properly estimated by potential consumers, and for various tool categories, is focused on different areas.

Although the most obvious effort seems to be the initial integration of the vulnerability scanning process (for live instances, code, or combination thereof), the process of verifying which of the results is REAL and EXPLOITABLE, to justify mitigation effort, may be just as tedious and even more time consuming.

To put the upcoming results into proper perspective, it's crucial to understand that the relative ratio presented in the various charts is exactly that - relative, and that in fact - most modern solutions are FAR BETTER in terms of accuracy then previous generations of tools (early DAST / early SAST / fuzzers / parsers). To further emphasize the perspective, assume the accuracy of modern tools falls in the following context when compared to that of previous generation tools:

* some MODERN solutions may do much better than in the chart generalizations (and some obviously worse)

And now, when the relative scale has been clarified, we can begin to compare modern technologies against each other.

Ease of Integration and Maintenance

Integrating a vulnerability detection solution may require different steps, depending on the complexity of the application being assessed.

The requirements will often include defining the scope of the test, providing necessary scan data (credentials / etc), configuring a scan policy, deploying an agent, etc.

The following table describes and assigns various integration/maintenance prerequisites to the relevant solution categories (DAST / SAST / IAST):

	DAST	SAST	Passive IAST	Active IAST
Deploy Application Deploy application Instance or scan an existing instance Common	✔ Required	X Not Required	✔ Required	✔ Required
Configure App URL Explicitly define scan target in scan policy/configuration Common	✔ Define URL in Scan Policy	✔ Source Control Path	? Licensing Purposes	✔ Define URL in Scan Policy
Define Credentials Define credentials / login scenarios in the scan policy Common	✔ Define Credentials AND Login Sequence	? Define Source Control Credentials (Optional)	X Not Required	✔ Define Credentials AND Login Sequence
Install an Agent Install a “scan” agent on the tested system framework Common	X Not Required	X Not Required	✔ Required	✔ Required
Define Scan Exclusions Configure “forbidden” pages and parameters to scan (logout, delete, etc) Situational	✔ Possible, Policy Specific	X Not Required	X Not Required	✔ Possible, Policy Specific
Handle Scan Barriers Configure/flag anti-CSRF parameters, custom tokens, complex login scenarios (micro-services), or record multi-phase processes Situational	✔ Possible, Policy Specific	X Not Required	X Not Required	✔ Possible, Policy Specific
Create Test Policies Define test plugins, "record" traffic, set threads, custom pages, and input vectors  Situational	✔ Possible, Policy Specific	X Not Required	? Coverage Optimization	✔ Possible, Policy Specific
Aggregate Sources Aggregate sources from key homegrown libraries (fragmented source projects) Situational	X Not Required	✔ Possible, Policy Specific	X Not Required	X Not Required
Maintain Policies Update scan policies in major changes in relatively complex applications Situational	✔ Possible, Policy Specific	? Source Code Path / Code Fragmentation	? Agent Re-install	✔ Possible, Policy Specific

To simplify the analysis of the data, we will present the integration / maintenance requirements in charts describing two main scenarios:
1) A typical scan of an unauthenticated OR common application with FORM/HTTP authentication,
2) A scan of a (relatively) complex application, with micro-service architecture, scan barriers (anti-CSRF mechanisms, multi-phase processes, etc), or similar complex prerequisites:

Click to Enlarge

Click to Enlarge

From the point of view of integration, some tools are easier to integrate than others, some have very little or no effort required for maintenance, and some require a specific scan policy in order to maximize the result efficiency.

To summarize the topic  :

• SAST - In an environment without any live application instances, SAST solutions can still be used to scan source code repositories, either directly or through the upload of source code projects, simplifying the initial assessment process, and making the integration of SAST relatively simple compared to alternatives. 
• Passive IAST - In an environment with live application instances, IAST solutions can be integrated simply by deploying an agent to the assessed application baseline framework. Although the initial integration may be difficult (objection from developers/QA, dedicated servers, configuration, potential performance issues in shared environments, etc), once the solution is set up there's very little maintenance .
• DAST / Active IAST - In an environment with live application instances, DAST solutions can be easily used to scan unauthenticated applications and will require minor configuration to scan applications with FORMS/HTTP authentication. More complex authentication methods, scan barriers (e.g. anti-CSRF mechanisms) or diverse architectures (micro-service architecture, REST, WS, etc) may require the creation of a dedicated policy and/or manual crawling session recording, while in the case of Active IAST solutions, in addition to requiring all the DAST prerequisites, will also require deployment of agents to the various tested layers.

Or in short, SAST solutions are probably easier to integrate, DAST and Passive IAST compete in terms of ease of integration, and Active IAST typically requires more effort to integrate than other solution categories (but provides adequately accurate results, as seen in the next section)

Effort of Weeding Out False Positives

The relative ratio of false positives derives from a number of methods that can be used by each technology to verify that identified vulnerabilities are not false positives, in addition to the individual solution efficiency:

Click to Enlarge

The justification for the chart diversity of the various solution categories stems from the verification methods that can be used by each solution category:

	DAST	SAST	Passive IAST	Active IAST
Execution URL Client-Driven Exploitation Entry-Point-To-Vuln-Code	✔ Exploit URL / Payload	x / ✔ Framework Dependent	✔ Exploit URL	✔ Exploit URL and Payload
Execution CLI Command Line Exploitation CLI-Param-To-Vuln-Code	X CLI Not Supported	✔ CLI Entry Point Detected	? Theoretically Possible	? Only via Passive
Flow/Taint Analysis Track Sequence of Methods to Activate Vulnerable Code	? Irrelevant for Technology	x / ✔ Key-Word Dependent	✔ In Effect	✔ In Effect
Input Effect on Sink Track Live Input Effect on the Vulnerable Code	? Through Binary Methods	X Not Performed	✔ Commonly Used	✔ Commonly Used
Modified Input Effect Track Modified Input Effect on the Vulnerable Code	✔ Payload Effect Analysis	X Not Performed	X Not Performed	✔ Payload Effect Analysis
Execution POC Time Delay, External Access, Browser Effect, Response Diff	✔ Commonly Used	X Not Performed	X Not Performed	✔ Commonly Used
Exploitation POC Full Scale Exploitation: Data Extraction, RCE, Shell Upload	✔ In Some Solutions	X Not Performed	X Not Performed	✔ In Some Solutions

Additional importance is given to the information the various tools provide to a HUMAN trying to discern the relevance of the issues reported, and tool-set (if any) provided to "reproduce" or manually "verify" the identified security issues.

Furthermore, the false positive factor will become EVER MORE IMPORTANT with the increase in volume of scanned applications. Weeding out false positive from actual issues will require time and effort from a security expert, and any misinterpretations will cost even more for developers to mitigate.

To complete the picture for the various technologies, let's present the RELATIVE integration/maintenance effort vs. the *typical* false-positive effort required to identify actual issues throughout the analysis cycle:

Click to Enlarge

So there you have it, technology and coverage, integration and false positive ratio for various *typical* modern technologies, pitted against each other.

And if the choice still seems complicated, that's ok, it's because it is.

There’s always the exception A relative obsolete and un-maintained DAST vulnerability scanner, in which there is little or no effort to “verify” detected vulnerabilities, will fare no better, and probably much worse, than a typical SAST/Passive-IAST solution, in terms of the ratio of false positive identification. On the other hand, a relatively immature or un-maintained SAST/Passive-IAST solution will fare much worse than presented in the charts - even in the effort required for integration and maintenance, especially compared to a modern DAST implementation.

And what of other aspects ?
The ability to detect common and exotic issues, the performance scale, the RISKS applied to using each technology in different environments, and the overall recommendation ?

Part II Coming Soon...