Tension and disconnects between software and systems engineering functions are not new. Grady Campbell wrote in 2004 that "systems engineering and software engineering need to overcome a conceptual incompatibility (physical versus informational views of a system)" and that systems engineering decisions can create or contribute to software risk if they "prematurely over-constrain software engineering choices" or "inadequately communicate information, including unknowns and uncertainties, needed for effective software engineering." This tension holds true for Department of Defense (DoD) programs as well, which historically decompose systems from the system level down to subsystem behavior and breakdown work for the program based on this decomposition. Hardware-focused views are typically deemed not appropriate for software, and some systems engineers (and most systems engineering standards) have not yet adopted an integrated view of the two disciplines. An integrated view is necessary, however, because in complex software-reliant systems, software components often interact with multiple hardware components at different levels of the system architecture. In this blog post, I describe recently published research conducted by me and other members of the SEI highlighting interactions on DoD programs between Agile software-development teams and their systems engineering counterparts in the development of software-reliant systems.

Foundations of Our Research

In the last several years, the DoD has focused efforts on decreasing the length of time needed to bring new software and other technical capabilities to soldiers. To accomplish this goal, members of the DoD acquisition community are increasingly turning their attention to Agile and other iterative development methods. The DoD 5000 series and other guidance for acquisition programs still offer a system-oriented perspective on acquisition. However, they do not provide strong guidance on how to leverage iterative software development methods within the greater context of iterative development methods.

With this research, our team--which also includes Suzanne Miller, Mary Ann Lapham, and Timothy A. Chick-does not advocate a particular development method. Instead, we explore the ways in which Agile software development teams are engaging systems engineers and associated stakeholders to identify factors that will help the DoD benefit from Agile methods and barriers to achieving those results.

As detailed in our technical note on this research, Agile Software Teams: How They Engage with Systems Engineering on DoD Acquisition Programs, two key facets of systems engineering help us to understand why systems engineering is an important player in programs adopting Agile methods:

• The product side of systems engineering: Systems engineering has a key role in transforming the artifacts that communicate intent of the system as understanding of the system evolves.
• The service side of systems engineering: Systems engineering has an equally important role in communicating and coordinating important information about the evolving knowledge of the system among the many stakeholders, including technical staff, end users, and management. Systems engineers have a strong conflict resolution role when inevitable technical and programmatic conflicts arise among stakeholders.

When we analyzed these two facets, what emerged were two distinct possibilities of how the systems engineering community might take advantage of Agile methods.

On the product side, the incremental, iterative approach with heavy user involvement common to all Agile methods could be leveraged to increase the speed of development of key requirement and design artifacts needed to implement different mission or system threads. Some methods, such as test-driven development, could be incorporated into the activities of systems engineering to increase the connection between the two sides of the typical systems engineering V lifecycle.

On the service side, at the scale of a program that requires a separate systems engineering function, the coordination, communication, and conflict-resolution services that systems engineering provides could translate into a product owner surrogate role, a Scrum of Scrums facilitator role, or other specialty roles that show up in scaling approaches, such as the Scaled Agile Framework (SAFe), which provides an interactive knowledge base for implementing agile practices at enterprise scale.

Three Different Approaches to Systems Engineering with Agile

At a high level, we envisioned three different approaches, which we refer to as engagement models:

• Agile software engineering teams interacting with traditional systems engineering teams operating on that traditional systems engineering V model
• systems engineers acting as Agile team members
• systems engineering teams actually applying Agile methods to their own work and systems engineering functions

In the first of these approaches--where traditional systems engineering is being used, and the systems engineering team is interacting with an Agile software team without being members of that team--we observed that Agile software engineering teams were providing deliverables to the systems engineering function at the boundary between the systems engineering function and software functions. These deliverables can include code or documentation and work products to facilitate technical reviews. The Agile team engaged in its Agile practices up to that point, assembled what it needed to hand off to a systems engineering function, and then passed those things over those boundaries. So, the Agile team was free to operate in its iterative and incremental way until it handed everything over. The team then entered the systems engineering domain, and the systems engineering teams executed according to their plans and processes, typically, with the traditional V model.

As a result, some of the decisions typically made in an Agile software project--such as the selection high business value or high technical infrastructure value--could not always be made because the programs were bound by the manner in which the systems engineering function had allocated the requirements and defined the work packages. For those teams, we did not always see all the benefit we observed in what I would call a "pure" Agile space because they had to deal with this interaction.

In the second of these approaches, a systems engineer typically operates on a software Agile team in the role of product owner, who is the person in charge of requirements and their prioritization. The systems engineer would be involved in the prioritization of features and functions going into a particular sprint. This involvement enables the systems engineer to follow the testing of the features and functionalities, so that when work products come to the boundary between the software team and the systems engineering function, systems engineers know what is coming into test and evaluation. Just as importantly, the systems engineering team is prepared. There is a smooth transition as the work product flows from one part of the engineering process to the other.

With this approach, we observed a smoother transition as well as additional opportunities for making changes in the systems engineering ideas and designs. We attributed these benefits to early learning enabled by the incremental approach and more detailed involvement with software. Some of the software implementation can change some of the system designs, so the software team actually had a little more influence on the systems engineering products when systems engineers worked in this way.

In the third approach, where systems engineers apply Agile methods to their own work, they iterate requirements development, develop the baselines, establish the baselines, and establish the designs throughout the lifecycle. The biggest issue we observed with this approach is the translation of "working software," a fundamental tenet of Agile methods when applied to software, to an equivalent in systems engineering. We observed this more often in commercial IT organizations, where there is no significant hardware development component. We did, however, observe one large project that adopted Agile systems engineering methods across system, hardware, and software tasks.

Although we only found one project, we have seen indicators of increased interest in this approach. The International Council on Systems Engineering (INCOSE) established a working group on Agile and systems engineering. Also, The National Defense Industrial Association has established an Agile and systems engineering working group. While this approach is the most novel, our research team did observe instances in which that dichotomy between product and service common to systems engineering comes in to play. Systems engineering functions that recognize the service side were more prepared, because their vision isn't limited to product artifact transformation.

Through our surveys and interviews, we observed that the greatest opportunities for successful Agile implementations occurred when systems engineering teams were, at the very least, aware of and engaged with the Agile processes. More generally, we identified successes and challenges in the following key areas:

• Automation. Investment in automation can help to harmonize Agile with traditional constructs by streamlining the development of documentation deliverables and communication. Lack of automation frustrates cost- and resource-effective testing and hampers communication and transparency between software and systems engineering teams.
• Insight/oversight. The constant, systemic delivery of production-ready code by Agile software teams over short iterations and the use of metrics and tools such as burn-down and cumulative flow diagrams offer up frequent, regular windows to monitor the progress and quality of the system under development, but stakeholders have to understand and actually exercise those insight opportunities to derive value from them.
• Training. Teams that provide systems engineers, government program office personnel, and other stakeholders with Agile-specific training on a recurring basis have reported success with communicating and expectation setting. Government program office staff training for the various career fields involved in acquisitions (e.g., contracting, finance), however, is generally functionally oriented: respondents indicated that strict capability-based training for government personnel has hampered the ability of many program office team members to conceptualize the changes in how contracts and delivery orders must be structured to most effectively engage Agile development teams.
• Role of sponsors, advocates, and coaches. Sponsors, advocates, and coaches for Agile can be instrumental to teams undergoing change to meet operator needs and variations in processes and to ensure that culture can change to support Agile methods, processes, and techniques. When coaches established and delivered training on Agile, systems engineering process and software development became better intertwined, and decisions were made together instead of separately. Leadership advocacy sets expectations for communication and collaboration within and across organizations and provides support that allows Agile practitioners to explore program-specific tailoring of the processes and documents required by acquisition regulations.
• Pilot programs. Respondents who engaged in piloting before broader rollouts indicated that demonstrated cost and schedule savings and the repeated demonstration of functional code often made believers out of previously skeptical systems engineering teams. They also consistently reported that the demonstration of cost, schedule, quality, insight, or predictability improvements via pilot projects also garnered positive feedback from government program offices, which made it easier to secure leadership buy-in and sponsorship for expanding the use of Agile on future efforts.
• Stakeholder involvement. Continuous collaboration with stakeholders is often difficult to achieve in a DoD setting due to factors such as lack of experience and lack of availability due to operations tempo or lack of funding. This collaboration is also challenging due to the variety of stakeholders that may bring conflicting requirements and/or priorities to the table. Certification and accreditation processes were cited as the most particular pain point, due to constantly evolving requirements and the typical "black box" nature of the processes.
• Defining and evolving requirements. Agile practitioners accept that the unknowns at the inception of a program are a natural and expected phenomenon, rather than treating those uncertainties as weak points. Agile software practitioners still report a tendency in many acquisition programs to create detailed software requirements at the inception of the program when the system is decomposed and the initial WBS developed. When detailed requirements are placed on contract at the beginning of a program, they are typically accompanied by change control boards and engineering change proposal (ECP) processes that many Agile practitioners report to be cumbersome, time-consuming, and expensive to complete.
• Verification and validation. In respondent settings where test and evaluation (T&E) personnel were included explicitly as members of the development team, they reported that many of the T&E personnel found significant benefit in developing acceptance criteria for stories and creating acceptance test fragments for portions of the system being completed. They also gained significant insight into the architecture, quality attributes, and functioning of the system, all of which translated into better test readiness when independent test activities were undertaken. Organizational issues and changes to job expectations may hamper efforts to include T&E staff as full team members, but careful communication, coordination, and use of test resources can help alleviate some of these challenges.
• Aligning the program roadmap with Agile increments. Agile development teams have worked with programs in a number of ways to attempt to tailor the existing milestones and reviews to more closely align with the manner in which Agile teams deliver software functionality. Many respondents engaged in new system development reported being granted the flexibility of tailoring program milestones such as the preliminary design review (PDR) and critical design review (CDR) by engaging systems engineers and program offices in sprint and iteration reviews. Several respondents reported negotiating for requirements (expressed as capabilities to evolve under the Agile paradigm) and then time-boxing their Agile increments against CDR targets.

Looking Ahead

One of the most important aspects of an Agile implementation is the use of the continuous improvement mechanism of retrospectives. During our interviews, we noted a variety of answers to the question

If you could change one thing about Agile interaction with systems engineering, what would it be?

Several themes emerged from the responses, including

• improved systems engineering discipline understanding of what Agile methods entail
• improved communication and reporting of progress for Agile teams
• alignment of Agile with policies and practices
• systems engineering process adaptation for Agile
• cultural change necessary for the DoD to fully realize the promise of Agile

These retrospectives inform future areas of work, including the appropriate use of metrics for reporting and managing programs employing Agile software development and on DoD contract vehicles and provisions that support the use of these methods.

We welcome your feedback on our research.

Additional Resources

To download our technical note detailing this research, Agile Software Teams: How They Engage with Systems Engineering on DoD Acquisition Programs, please visit
https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=295943.

To download the technical note, Potential Use of Agile Methods in Selected DoD Acquisitions: Requirements Development and Management, please visit
https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=89158.

To view all of our recent publications regarding Agile adoption in the Department of Defense, please visit
https://www.sei.cmu.edu/research-capabilities/all-work/display.cfm?customel_datapageid_4050=21716.