In the first blog entry of this two part series on common testing problems, I addressed the fact that testing is less effective, less efficient, and more expensive than it should be. This second posting of a two-part series highlights results of an analysis that documents problems that commonly occur during testing. Specifically, this series of posts identifies and describes 77 testing problems organized into 14 categories; lists potential symptoms by which each can be recognized; potential negative consequences, and potential causes; and makes recommendations for preventing them or mitigating their effects.

Why Testing is a Problem

A widely cited study for the National Institute of Standards & Technology (NIST) reports that inadequate testing methods and tools annually cost the U.S. economy between $22.2 billion and $59.5 billion, with roughly half of these costs borne by software developers in the form of extra testing and half by software users in the form of failure avoidance and mitigation efforts. The same study notes that between 25 percent and 90 percent of software development budgets are often spent on testing.

Despite the huge investment in testing mentioned above, recent data from Capers Jones shows that the different types of testing are relatively ineffective. In particular, testing typically only identifies from one-fourth to one-half of defects, while other verification methods, such as inspections, are typically more effective. Inadequate testing is one of the main reasons that software is typically delivered with approximately 2 to 7 defects per thousand lines of code (KLOC). While this may seem like a negligible number, the result is that major software-reliant systems are being delivered and placed into operation with hundreds or even thousands of residual defects. If software vulnerabilities (such as the CWE/SAN Top 25 Most Dangerous Software Errors) are counted as security defects, the rates are even more troubling.

Overview of Different Types of Testing Problems

The first blog entry in this series covered the following general types of problems that are not restricted to a single kind of testing:

• test planning and scheduling problems
• stakeholder involvement and commitment problems
• management-related testing problems
• test organization and professionalism problems
• test process problems
• test tools and environments problems
• test communication problems
• requirements-related testing problems

The remainder of this second post focuses on the following six categories of problems, each restricted to one of the following types of testing:

• unit testing
• integration testing
• specialty engineering testing
• system testing
• system of system testing
• regression testing

Unit testing problems primarily occur during the testing of individual software modules, typically by the same person who developed it in the first place. Design volatility could be causing excessive iteration of the unit test cases, drivers, and stubs. Unit testing could suffer from a conflict of interest as developers naturally want to demonstrate that their software works correctly while testers should seek to demonstrate that software fails. Finally, unit testing could be poorly and incompletely performed because the developers think it is relatively unimportant.

Integration testing problems occur during the testing of a set of units integrated into a component, a set of components into a subsystem, a set of subsystems into a system, or a set of systems into a system of systems. Integration testing concentrates on verifying the interactions between the parts of the whole. One potential problem is the difficulty of localizing defects to the correct part once the parts have been integrated. A second potential problem is inadequate, built-in test software that could help locate the cause of any failed test. Finally, a third problem is the potential lack of availability of the correct (versions of the) parts to integrate.

Specialty engineering testing problems occur when an inadequate amount of specialized testing of various quality characteristics and attribute testing takes place. More specifically, these problems involve inadequate capacity, concurrency, performance, reliability, robustness (e.g., error and fault tolerance), safety, security, and usability testing. While these are the most commonly occurring types of specialty engineering testing problems, other types of specialty testing problems may also exist depending on which quality characteristics and attributes are important (and thus the type of quality requirements that have been specified).

System testing problems occur during system-level testing and often cannot be eliminated because of the very nature of system testing. At best, recommended solutions can only mitigate these problems. Is it hard to test an integrated system's robustness (support for error, fault, and failure tolerance) due to the challenges of triggering system-internal exceptions and tracing their handling. System-level testing can be hard because temporary test hooks have typically been removed so that one is testing the actual system to be delivered. As with integration testing problems, demonstrating that system tests provide adequate test coverage is hard because reaching a specific code (e.g., fault tolerance paths) by only using inputs to the black-box system is hard. Finally, there is often inadequate mission-thread-based testing of end-to-end capabilities because system-testing is often performed using use-case-based testing, which is typically restricted to interactions with only a single, primary, system-external actor.

System-of-Systems (SoS) testing problems are often the result of SoS governance problems (i.e., everything typically occurs at the system-level rather than SoS-level). For example, SoS planning may not adequately cover SoS testing. Often, no organization is made explicitly responsible for SoS testing. Funding is often focused at the system-level, leaving little/no funding for SoS testing. Scheduling is typically performed only at the individual system level, and system-level schedule-slippages make it hard to schedule SoS testing.

SoS requirements are also often lacking or of especially poor quality, making it hard to test the SoS against its requirements. The individual system-level projects rarely allocate sufficient resources to support SoS testing. Defects are typically tracked only at the system level, making it difficult to address SoS-level defects. Finally, there tends to be a lot of finger-pointing and shifting of blame when SoS testing problems arise and SoS testing uncovers SoS-level defects.

Note that a SoS almost always consists of independently governed systems that are developed, funded, and scheduled separately. SoS testing problems therefore do not refer to systems that are developed by a prime contractor or integrated by a system integrator, nor do they refer to subsystems developed by subcontractors or vendors.

Regression testing problems occur during the performance of regression testing, both during development and maintenance. Often, there is insufficient automation of regression testing, which makes regression testing too labor-intensive to perform repeatedly, especially when using an iterative- and incremental-development-cycle. This overhead is one of the reasons that regression testing may not be performed as often as it should be.

When regression testing is performed, its scope is too localized because software developers think that changes in one part of the system will not propagate to other parts, and thereby cause faults and failures. Low-level regression testing is commonly easier to perform than higher-level regression testing, which results in an over-reliance on low-level regression tests. Finally, the test resources created during development may not be delivered and thus may not be available to support regression testing during maintenance.

Addressing Test-type Specific Problems

For each testing problem described above, I have documented several types of information useful for understanding the problem and implementing a solution. This information will be appearing in an upcoming SEI technical report. As an example of what will appear in this report, the testing problem "Over-reliance on COTS Testing Tools" has been documented with the information described below

Description. Too few of the regression tests are automated.

Potential symptoms. Many or even most of the tests are being performed manually.

Potential consequences.

• Manual regression testing takes so much time and effort that it is not done.
• If performed, regression testing is rushed, incomplete, and inadequate to uncover sufficient number of defects.
• Testers are making an excessive number of mistakes while manually performing the tests.
• Defects introduced into previously tested subsystems/software while making changes may remain in the operational system.

Potential causes. Testing stakeholders (e.g., managers and the developers of unit tests) may mistakenly believe that performing regression testing is neither necessary nor cost effective because

• of the minor scope of most changes
• system testing will catch any inadvertently introduced integration defects
• they are overconfident that changes have not introduced any new defects

Testing stakeholders may also not be aware of the

• importance of regression testing
• value of automating regression testing

Other potential causes may include

• Automated regression testing may not be an explicit part of the testing process.
• Automated regression testing may not be incorporated into the Test and Evaluation Master Plan (TEMP) or System/Software Test Plan (STP).
• The schedule may contain little or no time for the development and maintenance of automated tests.
• Tool support for automated regression testing may be lacking (e.g., due to insufficient test budget) or impractical to use.
• The initially developed automated tests may not be maintained.
• The initially developed automated tests may not be delivered with the system/software.

Recommendations. Prepare by explicitly addressing automated regression testing in the project's

• TEMP or STP
• test process documentation (e.g., procedures and guidelines)
• master schedule
• work break down structure (WBS)

Enable solution of the problem by

• providing training/mentoring to the testing stakeholders in the importance and value of automated regression testing
• providing sufficient time in the schedule for automating and maintaining the tests
• providing sufficient funding to pay for automated test tools
• ensuring that adequate resources (staffing, budget, and schedule) are planned and available for automating and maintaining the tests

Perform the following tasks:

• Automate as many of the regression tests as is practical.
• Where appropriate, use commercially available test tools to automate testing.
• Ensure that both automated and manual test results are integrated into the same overall test results database so that test reporting and monitoring are seamless.
• Maintain the automated tests as the system/software changes.
• Deliver the automated tests with the system/software.
• When relevant, identify this problem as a risk in the project risk repository.

Verify that

• The test process documentation addresses automated regression testing.
• The TEMP / STP and WBS address automated regression testing.
• The schedule provides sufficient time to automate and maintain tests.
• A sufficient number of the tests have been automated.
• The automated tests function properly.
• The automated tests are properly maintained.
• The automated tests are delivered with the system/software.

Related problems. no separate test plan, incomplete test planning, inadequate test schedule, unrealistic testing expectations / false sense of security, inadequate test resources, inadequate test maintenance, over-reliance on manual testing, tests not delivered, inadequate test configuration management (CM)

Benefits of Using the Catalog of Common Testing Problems This analysis of commonly occurring testing problems--and recommended solutions--can be used as training materials to better learn how to avoid, identify, and understand testing problems and mitigate them. Like anti-patterns, these problem categories can be used to improve communication between testers and testing stakeholders. This list can also be used to categorize problem types for metrics collection. Finally, they can be used as a checklist when

• producing test plans and related documentations
• evaluating contractor proposals
• evaluating test plans and related documentation (quality control)
• evaluating as-performed test process (quality assurance)
• identifying test-related risks and their mitigation approaches

Future Work

The framework of testing problems outlined in this series is the result of more than three decades of experience in assessments and my involvement in numerous projects and discussions with testing subject matter experts. Even after all this time, however, several unanswered questions remain that I intend to be the subject of future study:

• Probabilities. Which of these problems occur most often? What is the probability distribution of these problems? Which problems tend to cluster together? Do different problems tend to occur with different probabilities in different application domains such as commercial versus governmental versus military and web versus information technology versus embedded systems, etc.)?
• Severities. Which problems have the largest negative consequences? What are the probability distributions of harm caused by each problem?
• Risk. Based on the above probabilities and severities, which of these problems cause the greatest risks? Given these risks, how should one prioritize the identification and resolution of these problems?

I am interested in turning my work on this topic thus far into an industry survey and perform a formal study to answer these questions. I welcome your feedback on my work to date in the comments section below.

Additional Resources

To view a presentation on this work, please view, Common Testing Problems: Pitfalls to Prevent and Mitigate, and the associated Checklist Including Symptoms and Recommendations, which were presented at the FAA Verification and Validation Summit 8 (2012) in Atlantic City, New Jersey on 10 October 2012.