search menu icon-carat-right cmu-wordmark

Posts by Donald Firesmith

System Resilience Part 2: How System Resilience Relates to Other Quality Attributes

System Resilience Part 2: How System Resilience Relates to Other Quality Attributes

• Blog
Donald Firesmith

To most people, a system is resilient if it continues to perform its mission in the face of adversity. In other words, a system is resilient if it continues to operate appropriately and provide required capabilities despite excessive stresses that can or do cause disruptions. System resilience is not an isolated quality attribute. As this post, the second in a series on system resilience, details, it is directly related to robustness, safety, cybersecurity, anti-tamper, survivability,...

Read More
System Resilience: What Exactly is it?

System Resilience: What Exactly is it?

• Blog
Donald Firesmith

Over the past decade, system resilience (a.k.a., system resiliency) has been widely discussed as a critical concern, especially in terms of data centers and cloud computing. It is also vitally important to cyber-physical systems, although the term is less commonly used in that domain. Everyone wants their systems to be resilient, but what does that actually mean? And how does resilience relate to other quality attributes, such as availability, reliability, robustness, safety, security, and survivability?...

Read More
Mission Thread Analysis Using End-to-End Data Flows - Part 2

Mission Thread Analysis Using End-to-End Data Flows - Part 2

• Blog
Donald Firesmith

The first blog post in this series provided an overview of the E2E Mission Thread Data Flow Analysis (EMDA) method, an approach that analyzes the flow of data as they traverse end-to-end mission threads through the architecture components of a system of systems. That post addressed relevant challenges that EMDA helps system and software architects face and outlined the work products produced by the method. This second blog post discusses the process used to create...

Read More
Mission Thread Analysis Using End-to-End Data Flows  - Part 1

Mission Thread Analysis Using End-to-End Data Flows - Part 1

• Blog
Donald Firesmith

Although the vast majority of military missions require the successful collaboration of multiple cyber-physical systems within an overall system of systems (SoS), almost all system and software architects work on programs developing or sustaining individual systems and subsystems. Often, they do not sufficiently understand the ramifications of how their system interoperates with these other systems to accomplish the overall mission. The lack of an end-to-end (E2E) mission thread analysis leads to numerous difficulties, such as...

Read More
Testing Concurrent Systems: Concurrency Defects, Testing Techniques, and Recommendations

Testing Concurrent Systems: Concurrency Defects, Testing Techniques, and Recommendations

• Blog
Donald Firesmith

Concurrency, which exists whenever multiple entities execute simultaneously, is a ubiquitous and an unavoidable fact of life in systems and software engineering. It greatly increases system and software complexity, which directly impacts testing. Concurrency leads to nondeterministic behavior and numerous types of concurrency defects that require specialized approaches to uncover. At the SEI, we are often called upon to review development planning documents including Test and Evaluation Master Plans (TEMPs) and Software Test Plans (STPs)....

Read More
Multicore and Virtualization Recommendations

Multicore and Virtualization Recommendations

• Blog
Donald Firesmith

The first post in this series introduced the basic concepts of multicore processing and virtualization, highlighted their benefits, and outlined the challenges these technologies present. The second post addressed multicore processing, whereas the third and fourth posts concentrated on virtualization via virtual machines (VMs) and containers (containerization), respectively. This fifth and final post in the series provides general recommendations for the use of these three technologies--multicore processing, virtualization via VMs, and virtualization via containers--including mitigating...

Read More
Virtualization via Containers

Virtualization via Containers

• Blog
Donald Firesmith

The first blog entry in this series introduced the basic concepts of multicore processing and virtualization, highlighted their benefits, and outlined the challenges these technologies present. The second post addressed multicore processing, whereas the third post concentrated on virtualization via virtual machines. In this fourth post in the series, I define virtualization via containers, list its current trends, and examine its pros and cons, including its safety and security ramifications....

Read More
Virtualization via Virtual Machines

Virtualization via Virtual Machines

• Blog
Donald Firesmith

This posting is the third in a series that focuses on multicore processing and virtualization, which are becoming ubiquitous in software development. The first blog entry in this series introduced the basic concepts of multicore processing and virtualization, highlighted their benefits, and outlined the challenges these technologies present. The second post addressed multicore processing. This third post concentrates on virtualization via virtual machines (VMs). Below I define the relevant concepts underlying virtualization via VMs, list...

Read More
Multicore Processing

Multicore Processing

• Blog
Donald Firesmith

The first blog entry in this series introduced the basic concepts of multicore processing and virtualization, highlighted their benefits, and outlined the challenges these technologies present. This second post will concentrate on multicore processing, where I will define its various types, list its current trends, examine its pros and cons, and briefly address its safety and security ramifications....

Read More
Multicore and Virtualization: An Introduction

Multicore and Virtualization: An Introduction

• Blog
Donald Firesmith

Multicore processing and virtualization are rapidly becoming ubiquitous in software development. They are widely used in the commercial world, especially in large data centers supporting cloud-based computing, to (1) isolate application software from hardware and operating systems, (2) decrease hardware costs by enabling different applications to share underutilized computers or processors, (3) improve reliability and robustness by limiting fault and failure propagation and support failover and recovery, and (4) enhance scalability and responsiveness through the...

Read More
Seven Recommendations for Testing in a Non-Deterministic World

Seven Recommendations for Testing in a Non-Deterministic World

• Blog
Donald Firesmith

In a previous post, I addressed the testing challenges posed by non-deterministic systems and software such as the fact that the same test can have different results when repeated. While there is no single panacea for eliminating these challenges, this blog posting describes a number of measures that have proved useful when testing non-deterministic systems....

Read More
The Challenges of Testing in a Non-Deterministic World

The Challenges of Testing in a Non-Deterministic World

• Blog
Donald Firesmith

Many system and software developers and testers, especially those who have primarily worked in business information systems, assume that systems--even buggy systems--behave in a deterministic manner. In other words, they assume that a system or software application will always behave in exactly the same way when given identical inputs under identical conditions. This assumption, however, is not always true. While this assumption is most often false when dealing with cyber-physical systems, new and even older...

Read More
Open System Architectures: When and Where to be Closed

Open System Architectures: When and Where to be Closed

• Blog
Donald Firesmith

By Donald Firesmith Principal Engineer Software Solutions Division Due to advances in hardware and software technologies, Department of Defense (DoD) systems today are highly capable and complex. However, they also face increasing scale, computation, and security challenges. Compounding these challenges, DoD systems were historically designed using stove-piped architectures that lock the Government into a small number of system integrators, each devising proprietary point solutions that are expensive to develop and sustain over the lifecycle. Although...

Read More
A Taxonomy of Testing: What-Based and When-Based Testing Types

A Taxonomy of Testing: What-Based and When-Based Testing Types

• Blog
Donald Firesmith

By Donald Firesmith Principal Engineer Software Solutions Division There are more than 200 different types of testing, and many stakeholders in testing--including the testers themselves and test managers--are often largely unaware of them or do not know how to perform them. Similarly, test planning frequently overlooks important types of testing. The primary goal of this series of blog posts is to raise awareness of the large number of test types, to verify adequate completeness of...

Read More
A Taxonomy of Testing

A Taxonomy of Testing

• Blog
Donald Firesmith

By Donald Firesmith Principal Engineer Software Solutions Division While evaluating the test programs of numerous defense contractors, we have often observed that they are quite incomplete. For example, they typically fail to address all the relevant types of testing that should be used to (1) uncover defects (2) provide evidence concerning the quality and maturity of the system or software under test, and (3) demonstrate the readiness of the system or software for acceptance and...

Read More
Four Types of Shift Left Testing

Four Types of Shift Left Testing

• Blog
Donald Firesmith

One of the most important and widely discussed trends within the software testing community is shift left testing, which simply means beginning testing as early as practical in the lifecycle. What is less widely known, both inside and outside the testing community, is that testers can employ four fundamentally-different approaches to shift testing to the left. Unfortunately, different people commonly use the generic term shift left to mean different approaches, which can lead to serious...

Read More
Using V Models for Testing

Using V Models for Testing

• Blog
Donald Firesmith

The verification and validation of requirements are a critical part of systems and software engineering. The importance of verification and validation (especially testing) is a major reason that the traditional waterfall development cycle underwent a minor modification to create the V model that links early development activities to their corresponding later testing activities. This blog post introduces three variants on the V model of system or software development that make it more useful to testers,...

Read More
Common Testing Problems: Pitfalls to Prevent and Mitigate

Common Testing Problems: Pitfalls to Prevent and Mitigate

• Blog
Donald Firesmith

In the first blog entry of this two part series on common testing problems, I addressed the fact that testing is less effective, less efficient, and more expensive than it should be. This second posting of a two-part series highlights results of an analysis that documents problems that commonly occur during testing. Specifically, this series of posts identifies and describes 77 testing problems organized into 14 categories; lists potential symptoms by which each can be...

Read More
Common Testing Problems: Pitfalls to Prevent and Mitigate

Common Testing Problems: Pitfalls to Prevent and Mitigate

• Blog
Donald Firesmith

A widely cited study for the National Institute of Standards & Technology (NIST) reports that inadequate testing methods and tools annually cost the U.S. economy between $22.2 and $59.5 billion, with roughly half of these costs borne by software developers in the form of extra testing and half by software users in the form of failure avoidance and mitigation efforts. The same study notes that between 25 and 90 percent of software development budgets are...

Read More
A Deeper Dive into the Method Framework for Engineering System Architectures

A Deeper Dive into the Method Framework for Engineering System Architectures

• Blog
Donald Firesmith

Engineering the architecture for a large and complex system is a hard, lengthy, and complex undertaking. System architects must perform many tasks and use many techniques if they are to create a sufficient set of architectural models and related documents that are complete, consistent, correct, unambiguous, verifiable, usable, and useful to the architecture's many stakeholders. This blog posting, the second in a two-part series, takes a deeper dive into the Method Framework for Engineering System...

Read More
The Method Framework for Engineering System Architectures

The Method Framework for Engineering System Architectures

• Blog
Donald Firesmith

Engineering the architecture for a large and complex system is a hard, lengthy, and complex undertaking. System architects must perform many tasks and use many techniques if they are to create a sufficient set of architectural models and related documents that are complete, consistent, correct, unambiguous, verifiable, and both usable by and useful to the architecture's many stakeholders. This blog posting, the first in a two-part series, presents the Method Framework for Engineering System Architectures...

Read More
The Need to Specify Requirements for Off-Nominal Behavior

The Need to Specify Requirements for Off-Nominal Behavior

• Blog
Donald Firesmith

In our work with acquisition programs, we've often observed a major problem: requirements specifications that are incomplete, with many functional requirements missing. Whereas requirements specifications typically specify normal system behavior, they are often woefully incomplete when it comes to off-nominal behavior, which deals with abnormal events and situations the system must detect and how the system must react when it detects that these events have occurred or situations exist. Thus, although requirements typically specify how...

Read More
A Collaborative Method for Engineering Safety- and Security-Related Requirements

A Collaborative Method for Engineering Safety- and Security-Related Requirements

• Blog
Donald Firesmith

Background: In our research and acquisition work on commercial and Department of Defense (DoD) programs, we see many systems with critical safety and security ramifications. With such systems, safety and security engineering are used to managing the risks of accidents and attacks. Safety and security requirements should therefore be engineered to ensure that residual safety and security risks will be acceptable to system stakeholders. The first post in this series explored problems with quality requirements...

Read More
Obstacles in Engineering Safety- and Security-Related Requirements, Second in a Three-Part Series

Obstacles in Engineering Safety- and Security-Related Requirements, Second in a Three-Part Series

• Blog
Donald Firesmith

Background: In our research and acquisition work on commercial and Department of Defense (DoD) programs, ranging from relatively simple two-tier data-processing applications to large-scale multi-tier weapons systems, one of the primary problems that we see repeatedly is that acquisitionand development organizations encounter the following three obstacles concerning safety- and security-related requirements:...

Read More
The Importance of Safety- and Security-related Requirements, First of a Three-Part Series

The Importance of Safety- and Security-related Requirements, First of a Three-Part Series

• Blog
Donald Firesmith

In our research and acquisition work on commercial and Department of Defense (DoD) programs ranging from relatively simple two-tier data-processing applications to large-scale multi-tier weapons systems , one of the primary problems that we see repeatedly is that requirements engineers tend to focus almost exclusively on functional requirements and largely ignore the so-called nonfunctional requirements, such as data, interface, and quality requirements, as well as technical constraints. Unfortunately, this myopia means that requirements engineers overlook...

Read More

Contact the Author

Donald Firesmith

SEI Digital Library

SEI Publications

Visit the SEI Digital Library to see other publications by Donald Firesmith

View publications