One of the most important and widely discussed trends within the software testing community is shift left testing, which simply means beginning testing as early as practical in the lifecycle. What is less widely known, both inside and outside the testing community, is that testers can employ four fundamentally-different approaches to shift testing to the left. Unfortunately, different people commonly use the generic term shift left to mean different approaches, which can lead to serious misunderstandings. This blog post explains the importance of shift left testing and defines each of these four approaches using variants of the classic V model to illustrate them.

The Consequences of Testing Late in the Lifecycle

For decades, it has been well known that defects are more difficult and expensive to fix the later they are found in the lifecycle. This phenomena is one reason why treating testing as a sequential phase at the end of waterfall development has long been viewed as a major pitfall of system and software testing. Examples of the harm caused by postponing testing include

• Testers may be less involved in initial planning, often resulting in insufficient resources being allocated to testing.
• Many requirements, architecture, and design defects are not uncovered and fixed until after significant effort has been wasted on their implementation.
• Debugging (including identifying, localizing, fixing, and regression testing defects) becomes harder as more software is produced and integrated.
• Encapsulation makes it harder to perform whitebox testing and to achieve high levels of code coverage during testing.
• There is less time to fix defects found by testing, thereby increasing the likelihood that they will be postponed until later increments or versions of the system, which creates a "bow wave" of technical debt that can sink projects if it grows too large.

These negative consequences of late testing increase development and maintenance costs, lead to missed deadlines and schedule delays, decrease quality due to residual defects, and generally lower project morale and job satisfaction.

Testing

Because the term testing means different things to different people, we need to ensure a common understanding of the word before discussing the four different types of shift left testing. Traditionally, people have decomposed verification into four different methods: analysis, demonstration, inspection, and testing. For many people, however, testing has become synonymous with quality assurance and the word has been expanded to include all four verification methods. Here and in my book, Common System and Software Testing Pitfalls: How to Prevent and Mitigate Them: Descriptions, Symptoms, Consequences, Causes, and Recommendations, I restrict the term testing to its traditional meaning:

placing an executable into a known pretest state in a known pre-test environment and stimulating it with known stimuli and then verifying whether the executable's resulting behavior and postconditions match those predicted by the test oracle (e.g., requirements or user expectations, architecture, or design).

In other words, a test is essentially an experiment to determine whether something (typically software or a system) executes correctly. Shift left testing does not mean the use of non-testing methods such as inspections (including audits, desk checking, Fagan-inspections, reviews, and walkthroughs), analyses (including static analysis of software) because we have always used these methods early in the development cycle. When developers and testers wish to include the other verification methods, they should use the terms shift left quality assurance or shift left verification.

Testing and the V Model

The classic V-model is a traditional way of graphically representing software engineering activities. As shown in Figure 1, the left side of the V represents requirements, architecture, design, and implementation whereas the right side represents integration and testing. Note that the word "system" in Figure 1 could mean either a pure software application or a system (or subsystem) consisting of software, hardware, and potentially other types of components such as data, documents, people, facilities, and devices.

The V-model has come under a great deal of significant - and justified - criticism in recent decades because it can easily be interpreted to imply a strict sequential waterfall development cycle that is inconsistent with a modern evolutionary (that is, incremental, iterative, concurrent, and time-boxed) development in projects that use Agile or DevOps approaches. The V model can also be interpreted as prohibiting a test-first approach or test-driven design (TDD). On the other hand, as long as system and software engineers remember its limitations and view it merely as notional and only showing logical relationships between development activities, the V model provides a succinct way to illustrate our descriptions of approaches to shift left testing. With this in mind, shift left essentially means moving testing to the left on the V.

The Four Methods of Shifting Testing to the Left

There are four basic ways to shift testing earlier in the lifecycle (that is, leftward on the V model). These can be referred to as traditional shift left testing, incremental shift left testing, Agile/DevOps shift left testing, and model-based shift left testing.

Traditional Shift Left Testing. As shown in Figure 2 below, traditional shift left moves the emphasis of testing lower down (and therefore slightly to the left) on the right hand side of the V. Instead of emphasizing acceptance and system level testing (e.g., UI testing with record and playback tools), traditional shift left concentrates on unit and integration testing (e.g., using API testing and modern test tools). The transition to traditional shift left testing has largely been completed.

Incremental Shift Left Testing. Few modern software-reliant projects use a strict waterfall development cycle. As shown in Figure 3 below, many projects developing large and complex software-reliant systems decompose development into a small number of increments (Vs) having correspondingly shorter durations. The shift left illustrated by the dashed red arrows occurs because parts of the single, large waterfall V model's types of testing (shown in gray) are shifted left to become increments of the corresponding types of testing in the smaller incremental V models. When each increment is also a delivery to the customer and operations, then incremental shift left testing shifts both developmental testing and operational testing to the left. Incremental shift left testing is popular when developing large, complex systems, especially those incorporating significant amounts of hardware. Like traditional shift left, the transition to incremental shift left has also been largely completed.

Agile/DevOps Shift Left Testing. As shown in Figure 4, Agile and DevOps projects have numerous short duration Vs (a.k.a., sprints) in lieu of a single or small number of V as in the previous two examples of shift left testing. These small Vs would also be modified if one or more early sprints are used to block out the basic requirements and architecture or if test-first and test-driven development (TDD) are being performed. The shift left occurs because the types of testing on the right sides of the earliest of these tiny Vs are to the left of the corresponding types of testing on right side of the larger V(s) they replace. While Figure 4 below appears remarkably the same for Agile and DevOps, Agile testing is typically restricted to developmental testing and does not include operational testing, which occurs once the system is placed into operation. The transition to Agile/DevOps shift left testing is currently popular and ongoing.

Model-Based Shift Left Testing. The previous forms of shifting testing left all concentrated on beginning the testing of software earlier in the development cycle. Waiting until software exists to begin testing, however, largely and unnecessarily limits the use of testing to uncovering coding defects. This delay is particularly disturbing because from 45 percent to 65 percent of defects are introduced in the requirements, architecture, and design activities.

As shown in Figure 5 below, model testing moves testing to the left side of the Vs by testing executable requirements, architecture, and design models. This shift enables testing to begin almost immediately, instead of waiting a long time (traditional), medium time (incremental), or a short time (Agile/DevOps) until software on the right side of the Vs is available to test. This trend is just beginning and will become increasingly popular as executable models and associated simulation/testing tools become more widely available.

These four different types of shift left testing form a historical progression with each one building on the ones that preceded it:

1. Traditional shift left testing corrects an overemphasis on UI-based system and acceptance testing.
2. Incremental shift left testing introduces incremental testing via an incremental development cycle.
3. Agile/DevOps shift left testing introduces continuous testing (CT) via an evolutionary lifecycle composed of many short duration sprints.
4. Model-Based shift left testing introduces the testing of executable requirements, architecture, and design models.

Conclusion

Shift left testing is a powerful and well-known trend within the software testing community that is largely intended to better integrate testing into the system/software engineering and thereby uncover defects earlier when they are easier and less expensive to fix. What is less well known, both within and outside of the testing community, is that there are four different ways to shift testing left and that these methods build upon each other to greatly improve the efficiency, effectiveness, and even the scope of testing.

Each of these four variants of shift left testing brings advantages, not least of which is finding defects earlier when they are easier and cheaper to fix. Each new shift left method has proved more powerful than the previous one, so it is reasonable to suspect that the model-based shift left testing may well be the most powerful of all because it helps to uncover requirements, architecture, and design defects before they are even implemented in software. Not surprisingly, the SEI is investigating the testing of executable architecture models developed using the Architecture Analysis and Design Language (AADL).

Additional Resources

To read Don Firesmith's blog post, Using V Models for Testing, please visit
/blog/using-v-models-for-testing/.

To read more about the Testing at the End anti-pattern, please see his most recent book, Common System and Software Testing Pitfalls: How to Prevent and Mitigate Them: Descriptions, Symptoms, Consequences, Causes, and Recommendations, which was published as part of the SEI Series in Software Engineering.