There are more than 200 different types of testing, and many stakeholders in testing--including the testers themselves and test managers--are often largely unaware of them or do not know how to perform them. Similarly, test planning frequently overlooks important types of testing. The primary goal of this series of blog posts is to raise awareness of the large number of test types, to verify adequate completeness of test planning, and to better guide the testing process. In the previous blog entry in this series, I introduced a taxonomy of testing in which 15 subtypes of testing were organized around how they addressed the classic 5W+2H questions: what, when, why, who, where, how, and how well. This and future postings in this series will cover each of these seven categories of testing, thereby providing structure to roughly 200 types of testing currently used to test software-reliant systems and software applications.
This blog entry covers the four, top-level subtypes of testing that answer the following questions:
What-based testing: What is being tested?
Object-Under-Test-based (OUT) testing
When-based testing: When is the testing being performed?
Built-in-Test (BIT) testing
After exploring the what-based and when-based categories of testing, this post presents a section on using the taxonomy, as well as opportunities for accessing it.
By Julien Delange Member of the Technical Staff Software Solutions Division
For decades, safety-critical systems have become more software intensive in every domain--in avionics, aerospace, automobiles, and medicine. Software acquisition is now one of the biggest production costs for safety-critical systems. These systems are made up of several software and hardware components, executed on different components, and interconnected using various buses and protocols. For instance, cars are now equipped with more than 70 electronic control units (ECUs) interconnected with different buses and require about 100 million source lines of code (SLOC) to provide driver assistance, entertainment systems, and all necessary safety features, etc. This blog post discusses the impact of complexity in software models and presents our tool that produces complexity metrics from software models.
While evaluating the test programs of numerous defense contractors, we have often observed that they are quite incomplete. For example, they typically fail to address all the relevant types of testing that should be used to (1) uncover defects (2) provide evidence concerning the quality and maturity of the system or software under test, and (3) demonstrate the readiness of the system or software for acceptance and being placed into operation. Instead, many test programs only address a relatively small subset of the total number of potentially relevant types of testing, such as unit testing, integration testing, system testing, and acceptance testing. In some cases, the missing testing types are actually performed (to some extent) but not addressed in test-related planning documents, such as test strategies, system and software test plans (STPs), and the testing sections of systems engineering management plans (SEMPs) and software development plans (SDP). In many cases, however, they are neither mentioned nor performed. This blog, post, the first in a series on the many types of testing, examines the negative consequences of not addressing all relevant testing types and introduces a taxonomy of testing types to help testing stakeholders understand--rather than overlook--them.
Department of Defense (DoD) systems are becoming increasingly software reliant, at a time when concerns about cybersecurity are at an all-time high. Consequently, the DoD, and the government more broadly, is expending significantly more time, effort, and money in creating, securing, and maintaining software-reliant systems and networks. Our first post in this series provided an overview of the SEI's five-year technical strategic plan, which aims to equip the government with the best combination of thinking, technology, and methods to address its software and cybersecurity challenges. This blog post, the second in the series, looks at ongoing and new research we are undertaking to address key cybersecurity, software engineering and related acquisition issues faced by the government and DoD.
Object-oriented programs present considerable challenges to reverse engineers. For example, C++ classes are high-level structures that lead to complex arrangements of assembly instructions when compiled. These complexities are exacerbated for malware analysts because malware rarely has source code available; thus, analysts must grapple with sophisticated data structures exclusively at the machine code level. As more and more object-oriented malware is written in C++, analysts are increasingly faced with the challenges of reverse engineering C++ data structures. This blog post is the first in a series that discusses tools developed by the Software Engineering Institute's CERT Division to support reverse engineering and malware analysis tasks on object-oriented C++ programs.
1. Team coordination 2. Architectural runway 3. Align development and decomposition. 4. Quality-attribute scenarios 5. Test-driven development
This post presents the remaining five technical best practices, as well as three conditions that will help organizations achieve the most value from these recommended practices. This post was originally published in its entirety on the SPRUCE website.
Each year, the CERT Division of the SEI collaborates with CSO Magazine to develop a U.S. State of Cybercrime report1. These reports are based on surveys of more than 500 organizations across the country, ranging in size from fewer than...