From the braking system in your automobile to the software that controls the aircraft that you fly in, safety-critical systems are ubiquitous. Showing that such systems meet their safety requirements has become a critical area of work for software and systems engineers. "We live in a world in which our safety depends on software-intensive systems," editors of IEEE Software wrote in the magazine's May/June issue. "Organizations everywhere are struggling to find cost-effective methods to deal with the enormous increase in size and complexity of these systems, while simultaneously respecting the need to ensure their safety." The Carnegie Mellon Software Engineering Institute (SEI) is addressing this issue with a significant research program into assurance cases. Our sponsors are regularly faced with assuring that complex software-based systems meet certain kinds of requirements such as safety, security, and reliability. In this post, the first in a series on assurance cases and confidence, I will introduce the concept of assurance cases and show how they can be used to argue that a safety requirement (or other requirement such as security) has been met.

Making a Sound Evaluation

A system is deemed "safety-critical" when its failure could result in loss of life or catastrophic damage. Such systems are expensive to develop, build, and deploy. Due to the severe consequences of failure, most safety-critical systems are also subject to evaluation by some external authority (such as the U.S. Food and Drug Administration (FDA) for medical devices in the United States or the Ministry of Defence for flight control systems in the United Kingdom). To enable the evaluator to make a sound evaluation, developers must provide information about the system and its development, including requirements, documentation of design decisions, hazard analyses, development process, and verification and validation results. Developers submit this information (which constitutes an "argument") that they believe will convince evaluators to approve the system.

Industry has long relied on process-based arguments to help evaluators conduct their evaluations, but as the National Research Council of the National Academy of Science reported in 2007:

For a system to be regarded as dependable, concrete evidence must be present that substantiates the dependability claim. This evidence will take the form of a dependability case arguing that the required properties follow from the combination of the properties of the system itself (that is, the implementation) and the environmental assumptions. ... In addition, the case will inevitably involve appeals to the process by which the software was developed.

Although called a "dependability case" in the NRC report, most researchers, evaluators, and users have since coalesced around the term "assurance case." When developing an assurance case for safety, the case is usually called a "safety case." As defined by the U.K. Ministry of Defence:

A safety case is a structured argument, supported by a body of evidence that provides a compelling, comprehensible and valid case that a system is safe for a given application in a given environment.

An assurance case is somewhat similar in form to a legal case. In a legal case, there are two basic elements:

• evidence, including witnesses, fingerprints, DNA, etc.
• an argument given by the attorneys as to why the jury should believe that the evidence supports (or does not support) the claim that the defendant is guilty (or not guilty)

If an attorney argues that a defendant is guilty, but provides no evidence to support that argument, the jury would certainly have reasonable doubts about the guilt of the defendant. Conversely, if an attorney presents evidence, but no supporting argument explaining its relevance, the jury would have difficulty deciding how the evidence relates to the defendant.

The goal-structured assurance case is similar. There is evidence (such as test results) that a property of interest (such as safety) holds. Without an argument as to why the test results support the claim of safety, however, an interested party could have difficulty understanding its relevance or sufficiency. With only a detailed argument of system safety but no test results, it would again be hard to establish the system's safety.

A goal-structured assurance case therefore specifies a claim regarding a property of interest, evidence that supports that claim, and a detailed argument explaining how the evidence supports the claim. An assurance case differs from a legal case in one significant respect. In a legal case there are advocates for both sides (the defendant is guilty; the defendant is not guilty). Typically, an assurance case is only used to show one point of view--the system is safe (and not the system is unsafe.)

Figure 1 above shows a notional safety case in Goal Structuring Notation (GSN). The case argues that the system is safe because both of hazards A and B have been eliminated. The elimination of those hazards is supported by evidence Ev1, Ev2, and Ev3. The evidence might be test results, analysis, modeling, simulation, etc.

Support for the use of assurance cases in the United States is nascent but increasing:

• The FDA has taken steps towards requiring their use when a device manufacturer submits a medical device for approval.
• NASA suggested the use of assurance cases as a part of development of the Constellation system.
• The Aerospace Corporation used assurance cases to help with a GPS satellite ground station replenishment program.

In Europe and the United Kingdom, the use of assurance cases is more common. They are required in systems as diverse as flight control systems, nuclear reactor shutdown systems, and railroad signaling systems.

Whether required or not, creating an assurance case is not just a pro forma exercise. Assurance cases provide many benefits. For instance, the act of creating an assurance case helps stakeholders establish a shared understanding of issues that contribute to safety. Once complete, an assurance case provides a reviewable artifact documenting how safety is achieved in the system. The assurance case is also useful as an artifact that a reviewer or regulator can use to understand and make judgments about the system.

The SEI has an active research program in assurance cases, and in particular how to know that an assurance case adequately justifies its claim about the subject system, which is actually a classic philosophical problem:

determining the basis for belief in a hypothesis when it is impossible to examine every possible circumstance covered by the hypothesis

Our exploration has led us to examine theories from philosophy, law, rhetoric, mathematics, and artificial intelligence--as well as the vibrant assurance case community--as sources of relevant material to develop a theory of argumentation. The theory we are developing uses Baconian probability and eliminative induction to help eliminate sources of reduced confidence in a claim.

Subsequent posts in this series will discuss our research into what we are calling argumentation theory. The theory will help make it easier to construct, modify, and evaluate assurance cases that are believable and will also make it possible to determine how much justified confidence one should have in it. In the next post we will begin to discuss these subjects. In the meantime we welcome your comments on the topic of assurance cases below.

Additional Resources

To read the SEI technical report Toward a Theory of Assurance Case Confidence, please visit https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=28067

To read the SEI technical note Towards an Assurance Case Practice for Medical Devices, please visit
https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=8999

To read the report Software for Dependable Systems: Sufficient Evidence? please visit http://www.nap.edu/catalog/11923.html

To read the dissertation Arguing Safety--A Systematic Approach to Safety Case Management by Timothy Patrick Kelly, please visit
http://www-users.cs.york.ac.uk/~tpk/tpkthesis.pdf

To read the report Ministry of Defence, Defence Standard--0056: Safety Management Requirements for Defence Systems--Part 2, please visit
https://sts.defencegateway.mod.uk:443/Login.aspx?ReturnUrl=%2f%3fwa%3dwsignin1.0%26wtrealm%3dhttp%253a%252f%252fsso.defencegateway.mod.uk%252fadfs%252fservices%252ftrust%26wctx%3dead9c911-cb83-4500-8c70-5ef25a227886%26wct%3d2019-09-11T13%253a14%253a54Z&wa=wsignin1.0&wtrealm=http%3a%2f%2fsso.defencegateway.mod.uk%2fadfs%2fservices%2ftrust&wctx=ead9c911-cb83-4500-8c70-5ef25a227886&wct=2019-09-11T13%3a14%3a54Z