As part of an ongoing effort to keep you informed about our latest work, I'd like to let you know about some recently published SEI technical reports and notes. These reports highlight the latest work of SEI technologists in embedded systems, risk management, risk-based measurement and analysis, early lifecycle cost estimation, and techniques for detecting data anomalies. This post includes a listing of each report, author(s), and links where the published reports can be accessed on the SEI website.

Principles of Trust for Embedded Systems
By David Fisher
The development of trusted systems is a long-standing, elusive, and ill-defined objective in many domains. This paper gives substance and explicit meaning to the terms trust and trustworthy as they relate to automated systems and to embedded systems in particular. Principles of trust are identified. Some of their implications for software engineering practice and for the design of hardware-based trusted computing platforms are also discussed.
PDF Download

Mission Risk Diagnostic (MRD) Method Description
By Christopher Alberts, Julia Allen & Robert Stoddard
Although most programs and organizations use risk management when developing and operating software-reliant systems, preventable failures continue to occur at an alarming rate. In many instances, the root causes of these preventable failures can be traced to weaknesses in the risk management practices employed by those programs and organizations. In particular, the SEI's field experience indicates that programs and organizations throughout government and industry are unable to assess their risks effectively. For example, SEI independent assessments routinely uncover significant risks that have not been brought to the attention of key decision makers. When decision makers are unaware of significant risks, they are unable to take action to mitigate those risks. As a result, SEI researchers undertook a project to examine and improve the practice of risk assessment. The SEI has developed the Mission Risk Diagnostic (MRD) to assess risk in interactively complex, socio-technical systems across the life cycle and supply chain. To date, the SEI has employed the MRD in a variety of domains, including software acquisition and development, cybersecurity, software security, and business portfolio management. This technical note provides an overview of the MRD method.
PDF Download

CERT® Resilience Management Model Capability Appraisal Method (CAM) Version 1.1
By the Resilient Enterprise Management Team
The CERT® Resilience Management Model (CERT®-RMM), developed by the CERT® Program at the SEI, is the result of many years of research and development committed to helping organizations meet the challenge of managing operational risk and resilience in a complex world. In operational terms, resilience is an emergent property of an organization that can continue to carry out its mission after a disruption that does not exceed its operational limit.

The ability of an organization to assess its current level of capability using CERT-RMM as the reference model is essential for measuring the current competency of its operational practices, setting improvement targets, and establishing plans and actions to close any gaps.

The SEI has developed and maintained the Standard Capability Maturity Model® Integration (CMMI®) Appraisal Method for Process Improvement (SCAMPISM) family of appraisal methods from the CMMI product suite. Consultations with the SEI's CMMI program manager indicated that it would be appropriate to extend the pedigree of the SCAMPI family of appraisal methodologies for the CERT-RMM Capability Appraisal Method (CAM) Version 1.1.

This report demonstrates that the SCAMPI Version 1.2 method can be adapted and applied to CERT-RMM V1.1 as the reference model for a process appraisal.
PDF Download

CERT® Resilience Management Model (RMM) v1.1: Code of Practice Crosswalk Commercial Version 1.1
By Kevin G. Partridge & Lisa R. Young
CERT® Resilience Management Model (CERT-RMM) provides a reference model that allows organizations to make sense of their practice deployment in a process context. In this context, the primary goal of this document is to help model users and adopters to understand how CERT-RMM process areas, industry standards, and codes of practices that are used by organizations in an operational setting are connected. Additionally, this document helps to achieve a primary goal of CERT-RMM, which is to allow adopters to continue to use their preferred standards and codes of practice at a tactical level while maturing management and improvement of operational resilience at a process level. This document was also created with the objective to permit organizations to use CERT-RMM as a means for managing the complexities of deploying more than one standard or code of practice.
PDF Download

A Closer Look at 804: A Summary of Considerations for DoD Program Managers
By Stephany Bellomo
This report examines Section 804 National Defense Authorization Act (NDAA) for 2010 and related guidance documents through the lens of the Department of Defense (DoD) Information Technology (IT) Program Manager. The information in this report is intended to help the program manager reason about actions they may need to take to adapt and comply with the Section 804 NDAA for 2010 and associated guidance.
PDF Download

Risk-Based Measurement and Analysis: Application to Software Security
By Christopher Alberts, Julia Allen & Robert Stoddard
For several years, the software engineering community has been working to identify practices aimed at developing more secure software. Although some foundational work has been performed, efforts to measure software security assurance have yet to materialize in any substantive fashion. As a result, decision makers (e.g., development program and project managers, acquisition program offices) lack confidence in the security characteristics of their software-reliant systems. The CERT Program at the SEI has chartered the Software Security Measurement and Analysis (SSMA) Project to advance the state-of-the-practice in software security measurement and analysis. The SSMA Project is exploring how to use risk analysis to direct an organization's software security measurement and analysis efforts. The overarching goal is to develop a risk-based approach for measuring and monitoring the security characteristics of interactively complex software-reliant systems across the life cycle and supply chain. To accomplish this goal, the project team has developed the SEI Integrated Measurement and Analysis Framework (IMAF) and refined the SEI Mission Risk Diagnostic (MRD). This report is an update to the technical note, Integrated Measurement and Analysis Framework for Software Security (CMU/SEI-2010-TN-025), published in September 2010. This report presents the foundational concepts of a risk-based approach for software security measurement and analysis and provides an overview of the IMAF and the MRD.
PDF Download

Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE)
By Robert Ferguson, Dennis Goldenson, James McCurley, Robert W. Stoddard, David Zubrow, & Debra Anderson
Difficulties with estimating the costs of developing new systems have been well documented, and are compounded by the fact that estimates are now prepared much earlier in the acquisition lifecycle, before there is concrete technical information available on the particular program to be developed. This report describes an innovative synthesis of analytical techniques into a cost estimation method that models and quantifies the uncertainties associated with early lifecycle cost estimation.

The method described in this report synthesizes scenario building, Bayesian Belief Network (BBN) modeling and Monte Carlo simulation into an estimation method that quantifies uncertainties, allows subjective inputs, visually depicts influential relationships among program change drivers and outputs, and assists with the explicit description and documentation underlying an estimate. It uses scenario analysis and design structure matrix (DSM) techniques to limit the combinatorial effects of multiple interacting program change drivers to make modeling and analysis more tractable. Representing scenarios as BBNs enables sensitivity analysis, exploration of scenarios, and quantification of uncertainty. The methods link to existing cost estimation methods and tools to leverage their cost estimation relationships and calibration. As a result, cost estimates are embedded within clearly defined confidence intervals and explicitly associated with specific program scenarios or alternate futures.
PDF Download

An Investigation of Techniques for Detecting Data Anomalies in Earned Value Management Data
Mark Kasunic, James McCurley, Dennis Goldenson, & David Zubrow
Organizations rely on valid data to make informed decisions. When data integrity is compromised, the veracity of the decision-making process is likewise threatened. Detecting data anomalies and defects is an important step in understanding and improving data quality.

The study described in this report investigated statistical anomaly detection techniques for identifying potential errors associated with the accuracy of quantitative earned value management (EVM) data values reported by government contractors to the Department of Defense.

This research demonstrated the effectiveness of various statistical techniques for discovering quantitative data anomalies. The following tests were found to be effective when used for EVM variables that represent cumulative values: Grubbs' test, Rosner test, box plot, autoregressive integrated moving average (ARIMA), and the control chart for individuals. For variables related to contract values, the moving range control chart, moving range technique, ARIMA, and Tukey box plot were equally effective for identifying anomalies in the data.

One or more of these techniques could be used to evaluate data at the point of entry to prevent data errors from being embedded and then propagated in downstream analyses. A number of recommendations regarding future work in this area are proposed in this report.
PDF Download

Additional Resources:
For the latest SEI technical reports and papers, please visit
www.sei.cmu.edu/library/reportspapers.cfm