search menu icon-carat-right cmu-wordmark

Subject: Testing

Don't Play Developer Testing Roulette: How to Use Test Coverage

Don't Play Developer Testing Roulette: How to Use Test Coverage

• SEI Blog
Robert V. Binder

Suppose someone asked you to play Russian Roulette. Although your odds of surviving are 5 to 1 (83 percent), it is hard to imagine how anyone would take that risk. But taking comparable risk owing to incomplete software testing is a common practice. Releasing systems whose tests achieve only partial code coverage—the percentage of certain elements of a software item that have been exercised during its testing—is like spinning the barrel and hoping for the...

Read More
Testing Concurrent Systems: Concurrency Defects, Testing Techniques, and Recommendations

Testing Concurrent Systems: Concurrency Defects, Testing Techniques, and Recommendations

• SEI Blog
Donald Firesmith

Concurrency, which exists whenever multiple entities execute simultaneously, is a ubiquitous and an unavoidable fact of life in systems and software engineering. It greatly increases system and software complexity, which directly impacts testing. Concurrency leads to nondeterministic behavior and numerous types of concurrency defects that require specialized approaches to uncover. At the SEI, we are often called upon to review development planning documents including Test and Evaluation Master Plans (TEMPs) and Software Test Plans (STPs)....

Read More
Enabling Shift-Left Testing from Small Teams to Large Systems

Enabling Shift-Left Testing from Small Teams to Large Systems

• SEI Blog
Nanette Brown

Shift left is a familiar exhortation to teams and organizations engaged in Agile and Lean software development. It most commonly refers to incorporating test practices and an overall test sensibility early in the software development process (although it may also be applied in a DevOps context to the need to pull forward operations practices). Shift left sounds reasonably straightforward: just take the tasks that are on the right-hand side of your timeline and pull them...

Read More
Decision-Making Factors for Selecting Application Security Testing Tools

Decision-Making Factors for Selecting Application Security Testing Tools

• SEI Blog
Thomas Scanlon

In the first post in this series, I presented 10 types of application security testing (AST) tools and discussed when and how to use them. In this post, I will delve into the decision-making factors to consider when selecting an AST tool and present guidance in the form of lists that can easily be referenced as checklists by those responsible for application security testing....

Read More
10 Types of Application Security Testing Tools: When and How to Use Them

10 Types of Application Security Testing Tools: When and How to Use Them

• SEI Blog
Thomas Scanlon

Bugs and weaknesses in software are common: 84 percent of software breaches exploit vulnerabilities at the application layer. The prevalence of software-related problems is a key motivation for using application security testing (AST) tools. With a growing number of application security testing tools available, it can be confusing for information technology (IT) leaders, developers, and engineers to know which tools address which issues. This blog post, the first in a series on application security testing...

Read More
Seven Recommendations for Testing in a Non-Deterministic World

Seven Recommendations for Testing in a Non-Deterministic World

• SEI Blog
Donald Firesmith

In a previous post, I addressed the testing challenges posed by non-deterministic systems and software such as the fact that the same test can have different results when repeated. While there is no single panacea for eliminating these challenges, this blog posting describes a number of measures that have proved useful when testing non-deterministic systems....

Read More
The Challenges of Testing in a Non-Deterministic World

The Challenges of Testing in a Non-Deterministic World

• SEI Blog
Donald Firesmith

Many system and software developers and testers, especially those who have primarily worked in business information systems, assume that systems--even buggy systems--behave in a deterministic manner. In other words, they assume that a system or software application will always behave in exactly the same way when given identical inputs under identical conditions. This assumption, however, is not always true. While this assumption is most often false when dealing with cyber-physical systems, new and even older...

Read More
The Top 10 Blog Posts of 2015: Technical Debt, DevOps, Graph Analytics, Secure Coding, and Testing

The Top 10 Blog Posts of 2015: Technical Debt, DevOps, Graph Analytics, Secure Coding, and Testing

• SEI Blog
Douglas C. Schmidt

In 2015, the SEI blog launched a redesigned platform to make browsing easier, and our content areas more accessible and easier to navigate. The SEI Blog audience also continued to grow with an ever-increasing number of visitors learning more about our research in technical debt, shift-left testing, graph analytics, DevOps, secure coding, and malware analysis. In 2015 (from January 1 through December 15), the SEI blog logged 159,604 visits and sessions (we also switched analytics...

Read More
A Taxonomy of Testing: What-Based and When-Based Testing Types

A Taxonomy of Testing: What-Based and When-Based Testing Types

• SEI Blog
Donald Firesmith

By Donald Firesmith Principal Engineer Software Solutions Division There are more than 200 different types of testing, and many stakeholders in testing--including the testers themselves and test managers--are often largely unaware of them or do not know how to perform them. Similarly, test planning frequently overlooks important types of testing. The primary goal of this series of blog posts is to raise awareness of the large number of test types, to verify adequate completeness of...

Read More
A Taxonomy of Testing

A Taxonomy of Testing

• SEI Blog
Donald Firesmith

By Donald Firesmith Principal Engineer Software Solutions Division While evaluating the test programs of numerous defense contractors, we have often observed that they are quite incomplete. For example, they typically fail to address all the relevant types of testing that should be used to (1) uncover defects (2) provide evidence concerning the quality and maturity of the system or software under test, and (3) demonstrate the readiness of the system or software for acceptance and...

Read More
Testing, Agile Metrics, Fuzzy Hashing, Android, and Big Data: The SEI Blog Mid-Year Review (Top 10 Posts)

Testing, Agile Metrics, Fuzzy Hashing, Android, and Big Data: The SEI Blog Mid-Year Review (Top 10 Posts)

• SEI Blog
Douglas C. Schmidt

The SEI Blog continues to attract an ever-increasing number of readers interested in learning more about our work in agile metrics, high-performance computing, malware analysis, testing, and other topics. As we reach the mid-year point, this blog posting highlights our 10 most popular posts, and links to additional related resources you might find of interest (Many of our posts cover related research areas, so we grouped them together for ease of reference.) Before we take...

Read More
Four Types of Shift Left Testing

Four Types of Shift Left Testing

• SEI Blog
Donald Firesmith

One of the most important and widely discussed trends within the software testing community is shift left testing, which simply means beginning testing as early as practical in the lifecycle. What is less widely known, both inside and outside the testing community, is that testers can employ four fundamentally-different approaches to shift testing to the left. Unfortunately, different people commonly use the generic term shift left to mean different approaches, which can lead to serious...

Read More
Android, Heartbleed, Testing, and DevOps: An SEI Blog Mid-Year Review

Android, Heartbleed, Testing, and DevOps: An SEI Blog Mid-Year Review

• SEI Blog
Douglas C. Schmidt

In the first half of this year, the SEI blog has experienced unprecedented growth, with visitors in record numbers learning more about our work in big data, secure coding for Android, malware analysis, Heartbleed, and V Models for Testing. In the first six months of 2014 (through June 20), the SEI blog has logged 60,240 visits, which is nearly comparable with the entire 2013 yearly total of 66,757 visits. As we reach the mid-year point,...

Read More
The Importance of Automated Testing in Open Systems Architecture Initiatives

The Importance of Automated Testing in Open Systems Architecture Initiatives

• SEI Blog
Douglas C. Schmidt

To view a video of the introduction, please click here.The Better Buying Power 2.0 initiative is a concerted effort by the United States Department of Defense to achieve greater efficiencies in the development, sustainment, and recompetition of major defense acquisition programs through cost control, elimination of unproductive processes and bureaucracy, and promotion of open competition. This SEI blog posting describes how the Navy is operationalizing Better Buying Power in the context of their Open Systems...

Read More
Using V Models for Testing

Using V Models for Testing

• SEI Blog
Donald Firesmith

The verification and validation of requirements are a critical part of systems and software engineering. The importance of verification and validation (especially testing) is a major reason that the traditional waterfall development cycle underwent a minor modification to create the V model that links early development activities to their corresponding later testing activities. This blog post introduces three variants on the V model of system or software development that make it more useful to testers,...

Read More
Common Testing Problems: Pitfalls to Prevent and Mitigate

Common Testing Problems: Pitfalls to Prevent and Mitigate

• SEI Blog
Donald Firesmith

In the first blog entry of this two part series on common testing problems, I addressed the fact that testing is less effective, less efficient, and more expensive than it should be. This second posting of a two-part series highlights results of an analysis that documents problems that commonly occur during testing. Specifically, this series of posts identifies and describes 77 testing problems organized into 14 categories; lists potential symptoms by which each can be...

Read More
Common Testing Problems: Pitfalls to Prevent and Mitigate

Common Testing Problems: Pitfalls to Prevent and Mitigate

• SEI Blog
Donald Firesmith

A widely cited study for the National Institute of Standards & Technology (NIST) reports that inadequate testing methods and tools annually cost the U.S. economy between $22.2 and $59.5 billion, with roughly half of these costs borne by software developers in the form of extra testing and half by software users in the form of failure avoidance and mitigation efforts. The same study notes that between 25 and 90 percent of software development budgets are...

Read More
Improving Testing Outcomes Through Software Architecture

Improving Testing Outcomes Through Software Architecture

• SEI Blog
Paul Clements

Testing plays a critical role in the development of software-reliant systems. Even with the most diligent efforts of requirements engineers, designers, and programmers, faults inevitably occur. These faults are most commonly discovered and removed by testing the system and comparing what it does to what it is supposed to do. This blog posting summarizes a method that improves testing outcomes (including efficacy and cost) in a software-reliant system by using an architectural design approach, which...

Read More