Posted on by cybersecurityin
Thomas Scanlon Senior Cyber Security Engineer CERT Security Automation Directorate
In the first post in this series, I presented 10 types of application security testing (AST) tools and discussed when and how to use them. In this post, I will delve into the decision-making factors to consider when selecting an AST tool and present guidance in the form of lists that can easily be referenced as checklists by those responsible for application security testing.
Before looking at specific AST products, you should determine which class of AST tools is appropriate for your application. To accomplish this task, there are many factors that will help you decide which class, or type, of AST tools to use. I present some of the most prominent factors below along with supporting guidance for making a decision. While each factor is presented and discussed individually, the factors should all be considered collectively in relation to one another during the decision-making process.
Keep in mind that there is no one tool type addresses all issues, so in the long run a combination of tools is recommended. The factors here, however, are discussed mainly from the perspective of selecting a single AST tool to get started with security automation before additional tools are introduced into the environment. Most commonly, that first tool type used will be a static application security testing (SAST), dynamic application security testing (DAST), or origin analysis/software composition analysis (SCA) tool (the tools on the bottom of the pyramid in the figure below). Ultimately, the best decision is just to get started testing and use any AST tool, but some factors to help you make the most effective choice possible are discussed below.
AST Tool Type Decision Factors
Authorship in the context of application security testing refers to who develops the source code under evaluation. Typically, the source code is either developed in-house by the organization that will utilize the code or development is completely outsourced to a third-party. However, there are many other development scenarios such as when portions of code development are outsourced to several different third-parties and integrated together in-house. Knowing the authorship model can influence application testing strategies. When possible, it is a good idea to use both SAST and DAST tools regardless of authorship. However, when selecting a single tool type a starting point for testing, authorship can factor into decisions.
Development Model and Target Platform
The target platform refers to the environment and technical stack where the application will be deployed. Obviously, if the target platform is a mobile device, testing tools geared for mobile applications would be of most interest. Likewise, if the application is targeted for a cloud deployment, application security testing as a service (ASTaaS) may be attractive because it can easily be integrated with many cloud offerings. If an application will be deployed to the public Internet, DAST tools are recommended because the application will be exposed to black-hat attacks from potentially anyone on the Internet.
The integration-level factor is about the ability and feasibility of adding AST tools into the development life cycle. A general principle here is shift left--integrate AST tools as early as possible into the process.
Security processes or controls are often dictated by policy, contracts, and regulations. Some common examples are Risk Management Framework (RMF), Federal Information Security Management Act (FISMA), Health Insurance Portability and Accountability Act (HIPAA), Sarbanes-Oxley Act (SOX), payment-card industry (PCI) compliance, Control Objectives for Information and Related Technology (COBIT), and International Organization for Standardization (ISO) 9000. Some tools include mapping to these policies, contracts, and regulations.
Prior Findings or Incidents
Summary of Decision Factors for AST Tool Types
Examining each factor will allow you to build a list of AST tool types to consider. The factors are presented individually above, but will need to be considered in aggregate when making decisions. Some factors may push you to a certain tool type and other factors may push you away from that tool type. Ideally you will implement a combination of tools. SAST, DAST, and SCA should be used in combination whenever possible. IAST and hybrid tools can be used if needed to get the most coverage.
In cases where only one or two tool types can be considered, the decision factors above should help you prioritize what can be done. Keep in mind, however, that a strong understanding of traditional SAST, DAST, and SCA is useful for making decisions on MAST, IAST, and ASTaaS. Correlation, test-coverage, and ASTO tools can improve the performance and effectiveness of the other AST tool types, but are not usually the first types of tools to implement.
Decision Factors for AST Tool Selection
After you know which types of AST tools are desired for an application, actual tool selection is not much different from any other technology evaluation. Below are some factors to consider when selecting particular AST products.
Fortunately, there are a lot of free open-source AST tools, particularly SAST, DAST and SCA. It is a good idea to test your use of AST tools with open-source products before selecting a commercial vendor. However, commercial AST tool vendors offer tools that typically have more features and capabilities. If your budget allows only one or two commercial AST tools, you can supplement these with open-source tools.
IAST and hybrid tools may help you to control costs by combining features of SAST and DAST tools into one product, instead of buying and operating multiple products. ASTaaS can be expensive but may be cheaper in the long run and often make it easier to do budget calculations because these arrangements can typically be negotiated as flat-fee services with multiyear agreements.
Development Language and Technology Stack
Obviously, you need to select an AST tool that supports your development language, which helps to narrow the tool candidates. There are many options for popular languages such as Java, .NET, and C/C++. Look for tools that integrate with your developers' integrated development environments (IDEs). Some have native integration or robust APIs.
The large commercial AST tools support multiple languages, which may yield cost savings across projects. Some of the smaller AST tools support only one language but do so very well. If you use a niche language, however, your choices may be limited and you may have to buy an optional plug-in from commercial vendors.
There may be specific technical objectives described in application requirements, organizational policies, compliance directives, contracts, and similar guidance. Examples include controls related to known common vulnerabilities and exposures (CVEs), SQL injection, input validation and sanitization, buffer overflows, cross-site scripting, password management, ACLs and least privilege, etc. These are all things AST tools are designed to combat, but if you have specific requirements for certain aspects, you need to make sure your AST handles them appropriately. License adherence and management are two examples that SCA tools can help with.
Time and Human Resources
In the long run, incorporating AST tools into the development process should save time and effort on re-work by catching issues earlier. However, implementing AST tools requires some initial investment of time and resources.
AST tools can produce lots of results, and someone has to be able to manage and act on them. These tools also have a lot of knobs and buttons for calibrating the output, but it takes time to set them at a desirable level. Both false positives and false negatives can be troublesome if the tools are not set correctly.
Correlation and ASTO tools can help manage the remediation workflow and suppress issues as appropriate.
Wrapping Up and Looking Ahead
There are many types of tools available to perform automated application security testing. In keeping with the spirit of the defense in depth security adage, the more tools you run the more likely you are to reduce the weaknesses in your applications. It is costly and time consuming, however, to introduce tools into a development environment. We therefore recommend starting with one tool, becoming proficient with it, and then adding tools as time and resources allow. The guidance above is intended to help you in selecting an appropriate starting point for these activities.
This post and the preceding one have outlined a general introduction to application security testing tools. Future articles will cover more advanced topics such as how to perform automated testing in multi-factor authentication environments and how the usability of testing tools affects testing performance.
Learn about the National Institute of Standards and Technology (NIST) Software Assurance Metrics And Tool Evaluation (SAMATE) Project.
Learn about the Open Web Application Security Project (OWASP).
Learn about the SANS Institute.
Access and download the software, tools, and methods that the SEI creates, tests, refines, and disseminates.
Review the Department of Homeland Security (DHS) Build Security In website.