Posted on by Architecturein
Testing plays a critical role in the development of software-reliant systems. Even with the most diligent efforts of requirements engineers, designers, and programmers, faults inevitably occur. These faults are most commonly discovered and removed by testing the system and comparing what it does to what it is supposed to do. This blog posting summarizes a method that improves testing outcomes (including efficacy and cost) in a software-reliant system by using an architectural design approach, which describes a coherent set of architectural decisions taken by architects to help meet the behavioral and quality attribute requirements of systems being developed.
Developers of software-reliant systems must address several testing-related challenges. For example, testing is expensive and can account for more than 50 percent of a project's schedule and budget, depending on the criticality of the system. Unfortunately, some organizations assign a budget for testing and stop when that budget is consumed. Safety-critical software organizations also have a budget, but they often must reach a confidence level independent of the expenditures to meet certification standards. In either situation, improving the efficacy and cost of testing is essential to meeting requirements and business goals.
Another challenge is that few organizations inform the testing process by considering the software architecture, which comprises the structure of the software elements in a system, the externally visible properties of those elements, and the relationships among them. Ignoring or overlooking the software architecture during the testing process is problematic because the structures that comprise the software architecture ensure the quality attributes and enable the system to meet its requirements and business goals.
To address the challenges outlined above, we have developed an architectural design approach to testing software-reliant systems. The foundation of this approach involves creating testability profiles that give testers an actionable description of a design approach's effect on the testing practice. Each testability profile consists of four parts:
Testability profiles do not currently exist for architectural design approaches. Although pattern catalogs, such as the Pattern-Oriented Software Architecture (POSA) series and the "Gang of Four" book, are now common, there was a time when pattern descriptions were not widely available. If the cost and/or efficacy of testing can be substantially improved through the use of testability profiles, we might one day expect to see them documented alongside (or as part of) the pattern description of an approach.
To see how the testability profile and the architecture design approach fit together, suppose an engineer decided to build a service-oriented architecture (SOA) for a system. While that engineer has gained a lot of quality for the system, that system may now be susceptible to a class of faults specific to SOA. For example, the network may not send a service request the way that it should, or a particular service may not contain the quality attributes that are needed. After an architecture design approach testability profile is established, testers can decide whether to perform one or more of the following steps:
As a result of our research, testers will be able to determine the most important things to test for by illuminating new failure models that might not have been known before. Conversely, testers will also be able to determine failure models that they can safely assume will not occur. It is our hypothesis that this approach is broadly applicable to many types of systems. We are interested in working with organizations to pilot this approach, so if you would like us to consider your organization for our pilot program, please send an email to firstname.lastname@example.org.
For more information about our research in architecture support for testing, please visit