In today's systems it's very hard to know where systems end and software begins. Software performs an integrating function in many systems, often serving as the glue interconnecting other system elements. We also find that many of the problems in software systems have their roots in systems engineering, which is an interdisciplinary field that focuses on how to design and manage complex systems over their life cycles. For that reason, staff at the Carnegie Mellon University Software Engineering Institute (SEI) often conduct research in the systems engineering realm. Process frameworks, architecture development and evaluation methods, and metrics developed for software are routinely adapted and applied to systems. Better systems engineering supports better software development, and both support better acquisition project performance. This blog post, the latest in a series on this research, analyzes project performance based on systems engineering activities in the defense and non-defense industries.

The Case for Systems Engineering

An understanding of the value of systems engineering is necessary to justify a project's investment in systems engineering resources and activities. In 2010, the DoD encouraged the National Defense Industrial Association (NDIA) and the SEI to develop a stronger business case for systems engineering by expanding on a 2007 survey of systems developers. We responded by conducting a new survey in collaboration with not just the NDIA, but other professional organizations including the IEEE-Aerospace and Electronic Systems Society, and the International Council on Systems Engineering (INCOSE). For this latest study, we surveyed 148 individual projects at organizations developing systems to obtain answers to the following questions:

1. What systems engineering activities do you perform on your project?
2. How well does your project perform against cost, schedule, and technical objectives?
3. How challenging is your project?

Although most of the surveyed projects supply systems for the U.S. defense sector, we also received some responses from organizations serving other market sectors and operating in different countries.

We distributed a questionnaire that collected information from participating projects along three dimensions:

• systems engineering (SE) capability. We assessed systems engineering capabilities deployed on the project by investigating both the presence and the quality of work products resulting from SE activities. These work products were selected from those listed in the CMMI framework by a panel of SE experts. Based on this assessment, SE deployment for each project was categorized as low, medium, or high.
• project performance. We assessed project performance as a combination of cost performance (satisfaction of budget), schedule performance, and technical performance (satisfaction of requirements). Again, based on this assessment, project performance for each project was categorized as low, medium, or high.
• project challenge (PC). Some projects are inherently more challenging than others due to factors such as size, duration, technology maturity, interoperability, requirements, etc. Based on the combination of these factors, project challenge was categorized as low, or high.

In the 2012 report The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Study, we reported our initial results, which found a strong relationship between SE deployment and project performance. In particular, as projects applied more SE capabilities, they delivered better performance. For example, among those projects deploying the least SE, only 15 percent delivered the highest level of project performance. Among those deploying the most SE, however, 56 percent delivered the highest level of project performance.

As one would expect, our initial analysis also showed an inverse relationship between project challenge and project performance--more challenging projects do not perform as well as less challenging ones. But, we also learned that SE practices became even more valuable when used with these challenging projects. For the most challenging projects, the number of projects delivering high project performance increased from 8 percent to 62 percent with increased SE deployment. This result shows the increasing need for SE as projects become more challenging.

Defense vs. Non-Defense Projects

In our latest research, we revisited this data to study differences between projects in the defense and non-defense domains. We wanted to know if there were differences in the application of systems engineering, the performance of projects, and the effectiveness of the applied SE. Our aim with this most recent analysis was to identify best practices in one domain that could be transplanted to the benefit of the other domain.

As detailed in our recently published report on this latest research, The Business Case for Systems Engineering: Comparison of Defense-Domain and Non-Defense Projects, our latest analysis compared 75 defense projects with 32 non-defense projects from the fields of electronic equipment, communications, and health care. The 75 defense projects were a subset of the 116 defense projects responding to the 2012 survey. This subset was chosen to produce a sample of projects representing the same degree of project challenge as the non-defense projects.

We found that--when examining budget, schedule, and requirements satisfaction--performance against budget and technical requirements was the same in both defense and non-defense domains. With respect to schedule, however, non-defense projects performed somewhat better than did defense projects.

We examined the degree to which systems engineering was deployed among defense and non-defense projects. We measured deployment of SE practices on a scale of 1 (little SE deployment) to 4 (significant SE deployment). We also examined both the total SE applied to the project, and the SE applied in specific areas such as product architecture development, requirements, and verification. As shown in the figure below, we found that, for almost all specific areas, non-defense projects performed less systems engineering than defense projects.

When we analyzed the relationship between SE deployment and project performance, we found results similar to those of the initial study. In both the defense and non-defense domains, projects that deployed more SE delivered better performance. However, we also found that most of the relationships between SE deployment and project performance were stronger in the non-defense domain than they were in the defense domain. Thus, the non-defense projects seem to getting a larger return from the SE investments. This result is shown in the figure below.

Wrapping Up and Looking Ahead

The findings of our analysis are summarized as follows:

1. For both defense domain projects and non-defense projects, projects with better SE deployment deliver, on average, better project performance.
2. Non-defense projects deploy slightly less SE than defense projects
3. Non-defense projects deliver slightly better project performance than defense projects, primarily due to better adherence to schedule.
4. The strength of the relationships between SE deployment and project performance is stronger for non-defense projects than for defense projects.

The data we have collected does not provide any insight into the root causes of these differences; it merely indicates a measurable difference. These differences may be the result of an evolutionary construct. Perhaps non-defense projects are adopting new development technologies more rapidly than defense projects. Or the difference could be attributed to the processes that defense projects follow. Further study of these questions may identify processes and practices that can be transplanted from the non-defense domain to the defense domain to produce improved performance and acquisition in the Department of Defense.

Looking ahead, I am interested in conducting further research with projects in non-defense domains to gain insight into how they deploy their systems engineering and to develop a more concrete understanding of why non-defense projects are more effective in their systems engineering performance, with respect to schedule.

We welcome your feedback on this research in the comments section below.

Additional Resources

The technical report The Business Case for Systems Engineering: Comparison of Defense-Domain and Non-Defense Projects, can be downloaded at
https://resources.sei.cmu.edu/asset_files/SpecialReport/2014_003_001_91760.pdf.

The technical report The Business Case for Systems Engineering may be downloaded at
https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=34061.