In the first post in this series, I presented 10 types of application security testing (AST) tools and discussed when and how to use them. In this post, I will delve into the decision-making factors to consider when selecting an AST tool and present guidance in the form of lists that can easily be referenced as checklists by those responsible for application security testing.

Before looking at specific AST products, you should determine which class of AST tools is appropriate for your application. To accomplish this task, there are many factors that will help you decide which class, or type, of AST tools to use. I present some of the most prominent factors below along with supporting guidance for making a decision. While each factor is presented and discussed individually, the factors should all be considered collectively in relation to one another during the decision-making process.

Keep in mind that there is no one tool type addresses all issues, so in the long run a combination of tools is recommended. The factors here, however, are discussed mainly from the perspective of selecting a single AST tool to get started with security automation before additional tools are introduced into the environment. Most commonly, that first tool type used will be a static application security testing (SAST), dynamic application security testing (DAST), or origin analysis/software composition analysis (SCA) tool (the tools on the bottom of the pyramid in the figure below). Ultimately, the best decision is just to get started testing and use any AST tool, but some factors to help you make the most effective choice possible are discussed below.

AST Tool Type Decision Factors

Authorship

Authorship in the context of application security testing refers to who develops the source code under evaluation. Typically, the source code is either developed in-house by the organization that will utilize the code or development is completely outsourced to a third-party. However, there are many other development scenarios such as when portions of code development are outsourced to several different third-parties and integrated together in-house. Knowing the authorship model can influence application testing strategies. When possible, it is a good idea to use both SAST and DAST tools regardless of authorship. However, when selecting a single tool type a starting point for testing, authorship can factor into decisions.

• If the application code was written solely or largely in house, SAST tools should be the first choice. SAST tools are the most robust and should be used whenever possible.
• If the application code was written by a third party and delivered as an executable, DAST tools should be the first choice. While all decision factors together will shape the final decision, these are the best choices if authorship is the only thing known about an application.
• In-house applications also lend themselves better to correlation, test-coverage analysis, and Application Security Testing Orchestration (ASTO) tools than do third-party executables.

Source-Code Availability

• If the source code is available, SAST tools are the best choice. As mentioned above, SAST tools are the most robust and should be used whenever possible. If the source code is not available, it is best to use DAST tools.
• If the source code is available and other factors permit, both DAST and SAST tools should be used. Interactive Application Security Testing (IAST) and hybrid tools become an option in this case too.
• If the application was written by a third-party and the source code is not available, fuzzing and negative-testing tools and techniques should be used in addition to traditional DAST tools.
• If the application was written by a third-party and the source code is available, consider running build-and-compare tools to verify that the delivered executables can be generated exactly the same from the delivered source code.

Third-Party Components

• If the application makes heavy use of third-party components and libraries or has them in critical areas, SCA tools are the first choice, even before SAST tools. SCA tools can find components and libraries that weren't thought to be in the application; for this reason, SCA tools should be used whenever possible.
• If the application makes little or no use of third-party components and libraries, use SAST tools as a first choice.
• If the application was written largely in house and made minor use of libraries, use SAST and SCA.
• If the application was written by a third-party and you are unsure of library usage, use SCA and DAST.

Development Model and Target Platform

The target platform refers to the environment and technical stack where the application will be deployed. Obviously, if the target platform is a mobile device, testing tools geared for mobile applications would be of most interest. Likewise, if the application is targeted for a cloud deployment, application security testing as a service (ASTaaS) may be attractive because it can easily be integrated with many cloud offerings. If an application will be deployed to the public Internet, DAST tools are recommended because the application will be exposed to black-hat attacks from potentially anyone on the Internet.

• If using a traditional waterfall software-development life cycle (SDLC), SAST tools fit well into this process. DAST tools also fit well.
• If using an Agile approach, possibly with DevOps, IAST and hybrid tools usually fit better because traditional stand-alone DAST and SAST tools can be too time intensive for the development cycle.
• In a continuous integration/continuous delivery (CI/CD) model, IAST and hybrid tools again fit better, but usually an SCA is most important in these development models if third-party components are used frequently.
• If the application is written by a third-party, the development model isn't as much of a decision factor, but you may want to choose ASTaaS for this reason.
• If Agile and/or CI/CD development models are used, and third-party components and libraries are known to be used, SCA becomes a must, as early in the development cycle as possible.

Integration Level

The integration-level factor is about the ability and feasibility of adding AST tools into the development life cycle. A general principle here is shift left--integrate AST tools as early as possible into the process.

• If tools can be integrated early in the process, SAST and SCA tools are highly recommended.
• If tools cannot be integrated early in the development cycle, use DAST tools along with fuzzing and negative-testing tools.
• If tools cannot be integrated into the development life cycle at all, consider ASTaaS.
• If the application is written by a third-party and targeted for the cloud, and if tools cannot be integrated into development, use ASTaaS focused on DAST and SCA.

Compliance

Security processes or controls are often dictated by policy, contracts, and regulations. Some common examples are Risk Management Framework (RMF), Federal Information Security Management Act (FISMA), Health Insurance Portability and Accountability Act (HIPAA), Sarbanes-Oxley Act (SOX), payment-card industry (PCI) compliance, Control Objectives for Information and Related Technology (COBIT), and International Organization for Standardization (ISO) 9000. Some tools include mapping to these policies, contracts, and regulations.

• If you don't think you have any standards to follow, consider lists such as OWASP Top 10, SANS Top 25, and CERT Coding Standards. Because many compliance directives involve storage and protection of data, database security scanners are particularly useful. Correlation and ASTO tools are also useful for reporting.
• If the application is written in house using a waterfall SDLC, and strict Health Insurance Portability and Accountability Act (HIPAA) adherence and reporting are needed, use multiple SAST with a correlation tool and a database-scanning tool.

Prior Findings or Incidents

• If you have historical knowledge of prior application weaknesses, you can use them to help inform your strategy for application software testing.
• If your application was previously exploited using a known vulnerability, make sure to use a strong SCA tool or possibly multiple tools.
• If manual code reviews are showing weak coding practices, implement SAST tools early in the development process.
• If the application is written in house and is mainly a web application, but you also build a mobile version that has received many customer complaints about crashes and errors, use SAST, DAST, and mobile application security testing (MAST) tools. Remember that crashes are gateways to further exploitation.

Summary of Decision Factors for AST Tool Types

Examining each factor will allow you to build a list of AST tool types to consider. The factors are presented individually above, but will need to be considered in aggregate when making decisions. Some factors may push you to a certain tool type and other factors may push you away from that tool type. Ideally you will implement a combination of tools. SAST, DAST, and SCA should be used in combination whenever possible. IAST and hybrid tools can be used if needed to get the most coverage.

In cases where only one or two tool types can be considered, the decision factors above should help you prioritize what can be done. Keep in mind, however, that a strong understanding of traditional SAST, DAST, and SCA is useful for making decisions on MAST, IAST, and ASTaaS. Correlation, test-coverage, and ASTO tools can improve the performance and effectiveness of the other AST tool types, but are not usually the first types of tools to implement.

Decision Factors for AST Tool Selection

After you know which types of AST tools are desired for an application, actual tool selection is not much different from any other technology evaluation. Below are some factors to consider when selecting particular AST products.

Budget

Fortunately, there are a lot of free open-source AST tools, particularly SAST, DAST and SCA. It is a good idea to test your use of AST tools with open-source products before selecting a commercial vendor. However, commercial AST tool vendors offer tools that typically have more features and capabilities. If your budget allows only one or two commercial AST tools, you can supplement these with open-source tools.

IAST and hybrid tools may help you to control costs by combining features of SAST and DAST tools into one product, instead of buying and operating multiple products. ASTaaS can be expensive but may be cheaper in the long run and often make it easier to do budget calculations because these arrangements can typically be negotiated as flat-fee services with multiyear agreements.

Development Language and Technology Stack

Obviously, you need to select an AST tool that supports your development language, which helps to narrow the tool candidates. There are many options for popular languages such as Java, .NET, and C/C++. Look for tools that integrate with your developers' integrated development environments (IDEs). Some have native integration or robust APIs.

The large commercial AST tools support multiple languages, which may yield cost savings across projects. Some of the smaller AST tools support only one language but do so very well. If you use a niche language, however, your choices may be limited and you may have to buy an optional plug-in from commercial vendors.

Technical Objectives

There may be specific technical objectives described in application requirements, organizational policies, compliance directives, contracts, and similar guidance. Examples include controls related to known common vulnerabilities and exposures (CVEs), SQL injection, input validation and sanitization, buffer overflows, cross-site scripting, password management, ACLs and least privilege, etc. These are all things AST tools are designed to combat, but if you have specific requirements for certain aspects, you need to make sure your AST handles them appropriately. License adherence and management are two examples that SCA tools can help with.

Time and Human Resources

In the long run, incorporating AST tools into the development process should save time and effort on re-work by catching issues earlier. However, implementing AST tools requires some initial investment of time and resources.

AST tools can produce lots of results, and someone has to be able to manage and act on them. These tools also have a lot of knobs and buttons for calibrating the output, but it takes time to set them at a desirable level. Both false positives and false negatives can be troublesome if the tools are not set correctly.

Correlation and ASTO tools can help manage the remediation workflow and suppress issues as appropriate.

Wrapping Up and Looking Ahead

There are many types of tools available to perform automated application security testing. In keeping with the spirit of the defense in depth security adage, the more tools you run the more likely you are to reduce the weaknesses in your applications. It is costly and time consuming, however, to introduce tools into a development environment. We therefore recommend starting with one tool, becoming proficient with it, and then adding tools as time and resources allow. The guidance above is intended to help you in selecting an appropriate starting point for these activities.

This post and the preceding one have outlined a general introduction to application security testing tools. Future articles will cover more advanced topics such as how to perform automated testing in multi-factor authentication environments and how the usability of testing tools affects testing performance.

Additional Resources

Learn about the National Institute of Standards and Technology (NIST) Software Assurance Metrics And Tool Evaluation (SAMATE) Project.

Learn about the Open Web Application Security Project (OWASP).

Learn about the SANS Institute.

Access and download the software, tools, and methods that the SEI creates, tests, refines, and disseminates.

Review the Department of Homeland Security (DHS) Build Security In website.