search menu icon-carat-right cmu-wordmark

Agile, Architecture Fault Analysis, the BIS Wassenaar Rule, and Computer Network Design: The Latest Research from the SEI

Douglas C. Schmidt

By Douglas C. Schmidt
Principal Researcher

As part of an ongoing effort to keep you informed about our latest work, I would like to let you know about some recently published SEI technical reports, technical notes, and white papers. These reports highlight the latest work of SEI technologists in Agile software development and Agile-at-scale, software architecture fault analysis, computer network design, confidence in system properties, and system-of-systems development as well as commentary from two CERT researchers on the Proposed BIS Wassenaar Rule. This post includes a listing of each report, author(s), and links where the published reports can be accessed on the SEI website.

Contracting for Agile Software Development in the Department of Defense: An Introduction
By Eileen Wrubel and Jon Gross

This technical note (TN), part of an ongoing Software Engineering Institute series on Agile in the Department of Defense (DoD), addresses effective contracting for Agile software development. Contracting officers do not receive career field education targeted at achieving successful outcomes with Agile software development methods. For the purposes of this TN, the SEI gathered data from program office team members, contractors, and contracting officers about the state of contracting activities involving Agile development. The authors conducted a series of interviews and mined past interviews and survey data on Agile software development to understand common questions and concerns and provide some real-world examples to address them. This TN offers a primer on Agile based on a contracting officer's goals, describes how program office teams need to support contracting efforts, and addresses common concerns about Agile and how those concerns can be mitigated in the contracting process.
Download a PDF of the Report

Enabling Incremental Iterative Development at Scale: Quality Attribute Refinement and Allocation in Practice
By Neil Ernst, Stephany Bellomo, Robert Nord, Ipek Ozkaya

Lengthy requirements, design, integration, test, and assurance cycles delay delivery, resulting in late discovery of mismatched assumptions and system-level rework. In response, development methods that enable frequent iterations with small increments of functionality, such as agile practices, have become popular. But such methods de-emphasize architectural analysis; they assume the emergence or existence of a stable architecture. Yet as the business goals and context evolve, the architecture must also change, which requires allocating increments of quality attribute requirements to iterations along with other business capabilities. Quality attribute requirements (also called nonfunctional requirements) are hard to separate into smaller increments since they often crosscut many aspects of the product. As a result, allocation is uneven since it is challenging to decompose them and understand their value. Working with quality attribute requirements in an incremental and iterative fashion involves solving two problems: separating high-level requirements into their constituent parts and allocating them to iterations to fulfill the requirement. Underpinning both problems is the need for measurements to show that the requirement is satisfied. This report describes industry principles and practices used to smooth the development of business capabilities and suggests some approaches to enabling large-scale iterative development, or "Agile at scale."
Download a PDF of the Report

Improving Quality Using Architecture Fault Analysis with Confidence Arguments
By Peter H. Feiler, Charles B. Weinstock, John B. Goodenough, Julien Delange, Ari Z. Klein, and Neil Ernst

This case study shows how an analytical architecture fault-modeling approach can be combined with confidence arguments to diagnose a time-sensitive design error in a control system and to provide evidence that proposed changes to the system address the problem. The analytical approach, based on the SAE Architecture Analysis and Design Language (AADL) for its well-defined timing and fault behavior semantics, demonstrates that such hard-to-test errors can be discovered and corrected early in the lifecycle, thereby reducing rework cost. The case study shows that by combining the analytical approach with confidence maps, we can present a structured argument that system requirements have been met and problems in the design have been addressed adequately--increasing our confidence in the system quality. The case study analyzes an aircraft engine control system that manages fuel flow with a stepper motor. The original design was developed and verified in a commercial model-based development environment without discovering the potential for missed step commanding. During system tests, actual fuel flow did not correspond to the desired fuel flow under certain circumstances. The problem was traced to missed execution of commanded steps due to variation in execution time.
Download a PDF of the Report

CND Equities Strategy
By Jonathan M. Spring and Edward Stoner

The goal of computer network defense (CND) is to protect the system and ensure the organization operates resiliently under strain. To succeed, the CND strategy must plan for adversary response to defensive measures. Part of this planning is strategic management of equities. In the security field, the word equities has a special meaning: information assets that provide an advantage over the adversary, especially where the owner is confident the adversary does not know of those assets. Examples of equities include a clandestine informant or details of adversary tools, tactics, or procedures (TTPs) that facilitate attribution, detection, or prevention. To destroy or burn equities is to render information assets useless by disclosure to the adversary. To defend a network, the CND operator must reveal some information. When an operator blocks a command and control (C2) domain name or IP address, the adversary will almost certainly notice that their tool stops communicating. Likewise with cleaning an infected system. So the question is not whether to reveal equities, but which equities to reveal and which to hold in reserve at what times to gain maximum advantage. We can use game theory to model this via information states, but this short paper will stick to a more friendly and informal discussion.
Download a PDF of the Report

Eliminative Argumentation: A Basis for Arguing Confidence in System Properties
By John B. Goodenough, Charles B. Weinstock, Ari Z. Klein

Assurance cases provide a structured method of explaining why a system has some desired property, for example, that the system is safe. But there is no agreed approach for explaining what degree of confidence one should have in the conclusions of such a case. This report defines a new concept, eliminative argumentation, which provides a philosophically grounded basis for assessing how much confidence one should have in an assurance case argument. This report will be of interest mainly to those familiar with assurance case concepts and who want to know why one argument rather than another provides more confidence in a claim. The report is also potentially of value to those interested more generally in argumentation theory.
Download a PDF of the Report

Comments on Bureau of Industry and Security (BIS) Proposed Rule Regarding Wassenaar Arrangement 2013 Plenary Agreements Implementation for Intrusion and Surveillance Items
By Allen Householder and Art Manion

Art Manion and Allen Householder recently submitted comments to the Department of Commerce Bureau of Industry and Security on their proposed rule regarding Wassenaar Arrangement 2013 Plenary Agreements Implementation: Intrusion and Surveillance Items. This paper presents their detailed comments.
Download a PDF of the Report

State of Practice Report: Essential Technical and Nontechnical Issues Related to Designing SoS Platform Architectures
By Sholom G. Cohen and John Klein

This report presents an analysis of the state of the practice in system-of-systems (SoS) development. SoS architectures, or blueprints for integrating multiple systems based on common software platforms, have been successful in many commercial environments. The report discusses technical issues related to SoS common platform development and adoption in the Department of Defense (DoD) and the nontechnical constraints that must be satisfied. The analysis is based on information captured from 12 interviews of leading SoS developers in the DoD and industry, applying a SoS definition from the literature to identify gaps between the current state and the desired end state. The results of the study show that while commercial and DoD developers follow different approaches, all organizations report nontechnical constraints as more challenging than technical issues. For the DoD, these include leadership changes, shifting political priorities, and difficulty in replacing suppliers. The report recommends further study of SoS planning and Agile approaches that better support incremental development; bridging the gap from SoS to system concerns so that system designers understand SoS concerns and can focus on their products in the context of the SoS; and documenting the platform at all software levels, including architecture views and component integration strategies.
Download a PDF of the Report

Additional Resources

For the latest publications on SEI research, please visit


This post has been shared 0 times.