Posted on by Testingin
By Donald Firesmith
Software Solutions Division
There are more than 200 different types of testing, and many stakeholders in testing--including the testers themselves and test managers--are often largely unaware of them or do not know how to perform them. Similarly, test planning frequently overlooks important types of testing. The primary goal of this series of blog posts is to raise awareness of the large number of test types, to verify adequate completeness of test planning, and to better guide the testing process. In the previous blog entry in this series, I introduced a taxonomy of testing in which 15 subtypes of testing were organized around how they addressed the classic 5W+2H questions: what, when, why, who, where, how, and how well. This and future postings in this series will cover each of these seven categories of testing, thereby providing structure to roughly 200 types of testing currently used to test software-reliant systems and software applications.
This blog entry covers the four, top-level subtypes of testing that answer the following questions:
After exploring the what-based and when-based categories of testing, this post presents a section on using the taxonomy, as well as opportunities for accessing it.
What-Based test types include all of the different types of testing that deal with the question "What is being tested?"
OUT-Based Test Types. The first subtype of what test types varies based on the type of object under test (OUT). Specifically, one can test
The following series of graphics document the different test types that are based on the type of object under test:
Model testing can be simultaneously decomposed two different ways: by the type of executable model under test and by the means by which the model is executed during testing. Specifically, one can potentially test any executable requirements, architecture, design, or test model including models written in either graphical or textual modeling languages. Typically during testing, these models are executed by an execution or simulation engine running on a computer. However, if such tools to execute the models and tests are unavailable, a person can manually execute them.
There are numerous types of hardware testing. Under OUT-based testing, five of the most commonly used hardware testing types associated with computer hardware are continuity testing, hardware stress testing, highly accelerated life testing, hardware qualification testing, and power-off testing:
In addition to software, systems may also include hardware, data, documentation, personnel, manual procedures, facilities, and equipment. System testing includes subsystem testing, system integration testing, system testing, System of Systems (SoS) integration testing, and SoS testing. Unfortunately, the term system integration testing is sometimes used for testing the integration of subsystems into systems and sometimes for testing during integration of systems into systems of systems. System integration testing can be further decomposed into hardware-in-the-loop testing, human-in-the-loop testing, model in the loop testing, processor-in-the-loop testing, and software-in-the-loop testing.
Data Center Testing refers to the testing associated with a data center rather than a single software application that may be housed in the data center. Configuration testing determines whether the hardware and software making up the data center are properly configured (for example, to ensure security). Failover and restore testing ensures that when hardware or software fails, then the appropriate hot, warm, or cold failover occurs and then is properly rolled back when the failed components are restored to their proper states. Integrated system testing is the testing of a data center to ensure that its components have been properly installed, interoperate properly under normal and exceptional circumstances, and that supporting electrical and cooling subsystems function properly. Network traffic testing is the testing of one or more data center networks to determine whether the networks properly handle normal and excessive network traffic.
Testing of the software or system under test (SUT) can yield false positive and false negative results if there are defects in either the development tools, development environment(s), test tools, or test environment(s).
Domain-Based Testing. The second subtype of what test types varies based on whether the test type is domain-independent or domain-specific.
When-Based Test Types
When-Based Test Types include all of the different types of testing that primarily deal with the question "When is the testing being performed?"
Temporal-Order-Based Test Types. The first type of when-based testing varies based on the temporal order of the testing, which typically matches the order in which the objects under test are developed. Specifically, one can perform:
Lifecycle-Based Test Types. The first subtype of when test types varies based on the degree to which the associated lifecycle is evolutionary (i.e., incremental, iterative, and concurrent), which influences when and how often testing is performed. Specifically, one can perform:
Phase-Based Test Types. The second subtype of when test types varies based on the phase in which testing occurs. Specifically, one can perform:
Built-In-Test (BIT)-Specific Test Types are the direct incorporation of test software into the deliverable system under test. This third subtype of when test types vary based on when the BIT tests execute. Specifically, BIT can occur as the system is being powered up, continuously in the background during system operation, when an event (often a fault or failure) occurs that
Using the Taxonomy
Because different types of testing uncover different types of defects and because many types of testing (as well as other verification and validation methods) are needed to ensure an acceptably low amount of residual defects, it is important that all potentially relevant types of testing are considered. This taxonomy of testing types can be used as a checklist to ensure that test planning includes all appropriate testing types for the object under test and thereby ensures that no important test types are accidentally overlooked. Those responsible for test planning, as well as other testing stakeholders, can essentially spider their way down all of branches of the taxonomy's hierarchical tree-like structure, constantly considering the questions "Is this type of testing relevant? Will using it be cost-effective? Do we have sufficient resources to do this type of testing? Should it be used across the entire system or only in certain parts of the architecture and under certain conditions?"
Another way the taxonomy can help is by enabling testers to use divide and conquer as a technique to attack the size and complexity of system and software testing in terms of the different types of testing. This taxonomy can help one see the similarities between related types of testing and make it easier to learn and remember the different types of testing.
Blog entries and conference presentations are too short to properly document such a rich ontology. For this reason, I am currently developing a wiki that clearly defines each testing type, describes its potential applicability and how to use it, discusses its pros and cons, states how it addresses each of the 5Ws and 2Hs.
In future blog posts, I will summarize the other major subclasses of this taxonomy. The next post in the series will explore the testing types in the taxonomy related to the questions where is the testing being performed and why is the testing being performed.
On September 9, 2015 from 12:30 to 2:30 p.m. ET, I will presented an SEI Webinar on A Taxonomy of Testing Types. To view the webinar, please click here.
In the meantime, I will be presenting this taxonomy at next month's 10th Annual FAA Verification and Validation Summit and giving a more detailed tutorial on it at the SEI's Software Solutions Conference 2015 at the Hilton Crystal City in Arlington, Va., on November 16-18. For more information about the conference, please click here.
I have authored several posts on testing on the SEI blog including a series on common testing problems, a post presenting variations on the V model of testing, and, most recently, shift-left testing.