Bugs and weaknesses in software are common: 84 percent of software breaches exploit vulnerabilities at the application layer. The prevalence of software-related problems is a key motivation for using application security testing (AST) tools. With a growing number of application security testing tools available, it can be confusing for information technology (IT) leaders, developers, and engineers to know which tools address which issues. This blog post, the first in a series on application security testing tools, will help to navigate the sea of offerings by categorizing the different types of AST tools available and providing guidance on how and when to use each class of tool.

See the second post in this series, Decision-Making Factors for Selecting Application Security Testing Tools.

Application security is not a simple binary choice, whereby you either have security or you don't. Application security is more of a sliding scale where providing additional security layers helps reduce the risk of an incident, hopefully to an acceptable level of risk for the organization. Thus, application-security testing reduces risk in applications, but cannot completely eliminate it. Steps can be taken, however, to remove those risks that are easiest to remove and to harden the software in use.

The major motivation for using AST tools is that manual code reviews and traditional test plans are time consuming, and new vulnerabilities are continually being introduced or discovered. In many domains, there are regulatory and compliance directives that mandate the use of AST tools. Moreover--and perhaps most importantly--individuals and groups intent on compromising systems use tools too, and those charged with protecting those systems must keep pace with their adversaries.

There are many benefits to using AST tools, which increase the speed, efficiency, and coverage paths for testing applications. The tests they conduct are repeatable and scale well--once a test case is developed in a tool, it can be executed against many lines of code with little incremental cost. AST tools are effective at finding known vulnerabilities, issues, and weaknesses, and they enable users to triage and classify their findings. They can also be used in the remediation workflow, particularly in verification, and they can be used to correlate and identify trends and patterns.

Guide to Application Security Testing Tools

This graphic depicts classes or categories of application security testing tools. The boundaries are blurred at times, as particular products can perform elements of multiple categories, but these are roughly the classes of tools within this domain. There is a rough hierarchy in that the tools at the bottom of the pyramid are foundational and as proficiency is gained with them, organizations may look to use some of the more progressive methods higher in the pyramid.

Static Application Security Testing (SAST)

SAST tools can be thought of as white-hat or white-box testing, where the tester knows information about the system or software being tested, including an architecture diagram, access to source code, etc. SAST tools examine source code (at rest) to detect and report weaknesses that can lead to security vulnerabilities.

Source-code analyzers can run on non-compiled code to check for defects such as numerical errors, input validation, race conditions, path traversals, pointers and references, and more. Binary and byte-code analyzers do the same on built and compiled code. Some tools run on source code only, some on compiled code only, and some on both.

Dynamic Application Security Testing (DAST)

In contrast to SAST tools, DAST tools can be thought of as black-hat or black-box testing, where the tester has no prior knowledge of the system. They detect conditions that indicate a security vulnerability in an application in its running state. DAST tools run on operating code to detect issues with interfaces, requests, responses, scripting (i.e. JavaScript), data injection, sessions, authentication, and more.

DAST tools employ fuzzing: throwing known invalid and unexpected test cases at an application, often in large volume.

Origin Analysis/Software Composition Analysis (SCA)

Software-governance processes that depend on manual inspection are prone to failure. SCA tools examine software to determine the origins of all components and libraries within the software. These tools are highly effective at identifying and finding vulnerabilities in common and popular components, particularly open-source components. They do not, however, detect vulnerabilities for in-house custom developed components.

SCA tools are most effective in finding common and popular libraries and components, particularly open-source pieces. They work by comparing known modules found in code to a list of known vulnerabilities. The SCA tools find components that have known and documented vulnerabilities and will often advise if components are out of date or have patches available.

To make this comparison, almost all SCA tools use the NIST National Vulnerability Database Common Vulnerabilities and Exposures (CVEs) as a source for known vulnerabilities. Many commercial SCA products also use the VulnDB commercial vulnerability database as a source, as well as some other public and proprietary sources. SCA tools can run on source code, byte code, binary code, or some combination.

Database Security Scanning

The SQL Slammer worm of 2003 exploited a known vulnerability in a database-management system that had a patch released more than one year before the attack. Although databases are not always considered part of an application, application developers often rely heavily on the database, and applications can often heavily affect databases. Database-security-scanning tools check for updated patches and versions, weak passwords, configuration errors, access control list (ACL) issues, and more. Some tools can mine logs looking for irregular patterns or actions, such as excessive administrative actions.

Database scanners generally run on the static data that is at rest while the database-management system is operating. Some scanners can monitor data that is in transit.

Interactive Application Security Testing (IAST) and Hybrid Tools

Hybrid approaches have been available for a long time, but more recently have been categorized and discussed using the term IAST. IAST tools use a combination of static and dynamic analysis techniques. They can test whether known vulnerabilities in code are actually exploitable in the running application.

IAST tools use knowledge of application flow and data flow to create advanced attack scenarios and use dynamic analysis results recursively: as a dynamic scan is being performed, the tool will learn things about the application based on how it responds to test cases. Some tools will use this knowledge to create additional test cases, which then could yield more knowledge for more test cases and so on. IAST tools are adept at reducing the number of false positives, and work well in Agile and DevOps environments where traditional stand-alone DAST and SAST tools can be too time intensive for the development cycle.

Mobile Application Security Testing (MAST)

The Open Web Application Security Project (OWASP) listed the top 10 mobile risks in 2016 as

1. improper platform usage
2. insecure data storage
3. insecure communication
4. insecure authentication
5. insufficient cryptography
6. insecure authorization
7. client code quality
8. code tampering
9. reverse engineering
10. extraneous functionality

MAST Tools are a blend of static, dynamic, and forensics analysis. They perform some of the same functions as traditional static and dynamic analyzers but enable mobile code to be run through many of those analyzers as well. MAST tools have specialized features that focus on issues specific to mobile applications, such as jail-breaking or rooting of the device, spoofed WI-FI connections, handling and validation of certificates, prevention of data leakage, and more.

Application Security Testing as a Service (ASTaaS)

As the name suggests, with ASTaaS, you pay someone to perform security testing on your application. The service will usually be a combination of static and dynamic analysis, penetration testing, testing of application programming interfaces (APIs), risk assessments, and more. ASTaaS can be used on traditional applications, especially mobile and web apps.

Momentum for the use of ASTaaS is coming from use of cloud applications, where resources for testing are easier to marshal. Worldwide spending on public cloud computing is projected to increase from $67B in 2015 to $162B in 2020.

Correlation Tools

Dealing with false positives is a big issue in application security testing. Correlation tools can help reduce some of the noise by providing a central repository for findings from others AST tools.

Different AST tools will have different findings, so correlation tools correlate and analyze results from different AST tools and help with validation and prioritization of findings, including remediation workflows. Whereas some correlation tools include code scanners, they are useful mainly for importing findings from other tools.

Test-Coverage Analyzers

Test-coverage analyzers measure how much of the total program code has been analyzed. The results can be presented in terms of statement coverage (percentage of lines of code tested) or branch coverage (percentage of available paths tested).

For large applications, acceptable levels of coverage can be determined in advance and then compared to the results produced by test-coverage analyzers to accelerate the testing-and-release process. These tools can also detect if particular lines of code or branches of logic are not actually able to be reached during program execution, which is inefficient and a potential security concern. Some SAST tools incorporate this functionality into their products, but standalone products also exist.

Since the functionality of analyzing coverage is being incorporated into some of the other AST tool types, standalone coverage analyzers are mainly for niche use.

Application Security Testing Orchestration (ASTO)

ASTO integrates security tooling across a software development lifecycle (SDLC). While the term ASTO is newly coined by Gartner since this is an emerging field, there are tools that have been doing ASTO already, mainly those created by correlation-tool vendors. The idea of ASTO is to have central, coordinated management and reporting of all the different AST tools running in an ecosystem. It is still too early to know if the term and product lines will endure, but as automated testing becomes more ubiquitous, ASTO does fill a need.

Selecting Testing Tool Types

There are many factors to consider when selecting from among these different types of AST tools. If you are wondering how to begin, the biggest decision you will make is to get started by beginning using the tools. According to a 2013 Microsoft security study, 76 percent of U.S. developers use no secure application-program process and more than 40 percent of software developers globally said that security wasn't a top priority for them. Our strongest recommendation is that you exclude yourself from these percentages.

There are factors that will help you to decide which type of AST tools to use and to determine which products within an AST tool class to use. It is important to note, however, that no single tool will solve all problems. As stated above, security is not binary; the goal is to reduce risk and exposure.

Before looking at specific AST products, the first step is to determine which type of AST tool is appropriate for your application. Until your application software testing grows in sophistication, most tooling will be done using AST tools from the base of the pyramid, shown in blue in the figure below. These are the most mature AST tools that address most common weaknesses.

After you gain proficiency and experience, you can consider adding some of the second-level approaches shown below in blue. For instance, many testing tools for mobile platforms provide frameworks for you to write custom scripts for testing. Having some experience with traditional DAST tools will allow you to write better test scripts. Likewise, if you have experience with all the classes of tools at the base of the pyramid, you will be better positioned to negotiate the terms and features of an ASTaaS contract.

The decision to employ tools in the top three boxes in the pyramid is dictated as much by management and resource concerns as by technical considerations.

If you are able to implement only one AST tool, here are some guidelines for which type of tool to choose:

• If the application is written in-house or you have access to the source code, a good starting point is to run a static application security tool (SAST) and check for coding issues and adherence to coding standards. In fact, SAST is the most common starting point for initial code analysis.
• If the application is not written in house or you otherwise don't have access to the source code, dynamic application security testing (DAST) is the best choice.
• Whether you have access to the source code or not, if a lot of third-party and open-source components are known to be used in the application, then origin analysis/software composition analysis (SCA) tools are the best choice. Ideally, SCA tools are run alongside SAST and/or DAST tools, but if resources only allow for implementation of one tool, SCA tools are imperative for applications with 3rd party components because they will check for vulnerabilities that are already widely known.

Wrapping Up and Looking Ahead

In the long run, incorporating AST tools into the development process should save time and effort on re-work by catching issues earlier. In practice, however, implementing AST tools requires some initial investment of time and resources. Our guidance presented above is intended to help you select an appropriate starting point. After you begin using AST tools, they can produce lots of results, and someone must manage and act on them.

These tools also have many knobs and buttons for calibrating the output, but it takes time to set them at a desirable level. Both false positives and false negatives can be troublesome if the tools are not set correctly.

In the next post in this series, I will consider these decision factors in greater detail and present guidance in the form of lists that can easily be scanned and used as checklists by those responsible for application security testing.

Additional Resources

Read the second post in this series, Decision-Making Factors for Selecting Application Security Testing Tools.

Learn about the National Institute of Standards and Technology (NIST) Software Assurance Metrics And Tool Evaluation (SAMATE) Project.

Learn about the Open Web Application Security Project (OWASP).

Learn about the SANS Institute.

Access and download the software, tools, and methods that the SEI creates, tests, refines, and disseminates.

Review the Department of Homeland Security (DHS) Build Security In website.