In a March 2019 report, the Defense Innovation Board (DIB)--a group of advisors focused on bringing the technical advantages employed by Silicon Valley to the Department of Defense (DoD)--noted that the United States faces threats that are evolving at an ever-increasing pace. The DIB also noted that the DoD's ability to adapt and respond to these threats is now determined by its ability to develop and deploy software to the field rapidly. As the DIB and other reports have noted, the DoD's current approach to software development is broken and a leading source of risk: "it takes too long, is too expensive, and exposes warfighters to unacceptable risk by delaying their access to tools they need to ensure mission success."
In this series of blog posts regarding sustainment, we will explore four areas of potentially significant impact that military software organizations must address to evolve from a traditional, waterfall software acquisition-and-development process to newer software development practices.

• software architecture techniques (addressed in this post)
• cybersecurity techniques (addressed in the second post in the series)
• Agile tools and methods (addressed in the third post in the series)
• software automation (addressed in the fourth and final post in this series)

These recommendations are based on lessons learned by SEI technical staff in their work with government customers, although they can be applied to pretty much any sustainment organization, both large and small. This series of four blogs will examine each of the four areas and explore some key challenges and recommendations.

For all large DoD weapon systems, after the development contract is finished, the systems go into the sustainment phase. Software sustainment involves coordinating the processes, procedures, people, information, and databases required to support, maintain, and operate software-reliant aspects of military systems. Federal regulations for sustainment classify work into two budget categories: a one-year budget for operations maintenance (O&M) and a two to three-year budget for research and development (RDT&E or R&D for short). But as my colleague Mike Phillips wrote in a recent blog post, during sustainment, systems have always had to correct defects while evolving to support changing requirements:

Software sustainment was initially viewed as limited to fixing errors with small patches, but the need for greater capability has grown, including major revisions in software design. Consequently, engineers have rethought the idea of conventional software maintenance. Today, they have evolved toward the process as continuous engineering, which lies at the intersection of fixing problems and providing new capabilities made possible by new hardware or new threats that demand new functionality. In this environment, software sustainment needed to become a well-developed capability in the government's software development centers.

The concept of modernization is generally more perfective (changes to the software to improve performance or add functionality) and preventative (changes to software intended to prevent cyber threats, improve maintainability, or reduce the cost of ownership). Since the approval process for R&D funding takes longer and only occurs every two years, perfective and preventive changes are often delayed. Modernization thus often becomes a large project, despite having to fit into the sustainment period.

The DoD typically sustains large, mission-critical systems for many decades. After these systems are fielded, and the organization relies on them, these systems start to change in multiple ways, as my colleague Robert Ferguson detailed in a recent blog post on defining software sustainment. When there are multiple instances (e.g., airplanes, ships, etc.), it is a hard to keep the implementations consistent across all systems in operation.

In support of its mission to maintain combat-ready forces, the DoD acquires, fields, and sustains large weapons platforms upon which it relies to be ready to operate when needed and to meet their expected service lives. Sustainment organizations in the military are currently responsible for the integration of sustainment activities to keep the platforms operational and the modernization planning to add new technologies.

Due to how these systems are used, this integration is a challenging task since the systems are usually in service and not physically located near the sustainment organization. In addition, it is necessary to manage both the hardware and the software baselines, which makes upgrades complicated and time-consuming.

The software architecture of a program or computing system is a depiction of the system that aids in understanding how the system will behave. An architecture cannot be maintained if it has not been sufficiently communicated. No matter how good the architecture is, it won't matter if it is not well described. As my former colleague Paul Clements wrote in his book Documenting Software Architectures: Views and Beyond

Architecture documentation helps architects make the right decisions; it tells developers how to carry them out; and it records those decisions to give a system's future caretakers insight into the architect's solution.

It is important to have knowledge of the architecture and keep it up to date because:

• An architecture will enable or inhibit a system's driving quality attributes.
• The decisions made in an architecture allow developers and organizations to reason about and manage change as the system evolves.
• A documented architecture enhances communication among stakeholders, as detailed in Software Architecture in Practice, Third Edition.

In this case, the scope of software development is exclusively legacy modernization and tech refresh. These activities span minor changes to more extensive modifications, although many of the projects are engaged in making minor changes that do not impact the architecture.

We identified the following three recommendations based on our DoD experience.

• The architecture documentation developed by the contractor should be delivered as part of the contract. Without the physical representation of the architecture (i.e., sufficient architecture documentation), it can be hard to assess whether a particular change impacts the architecture's ability to continue to satisfy requirements. It is possible to reconstruct the architecture; however, this is a tedious and time-consuming task. Given these constraints, it is justifiable for the additional cost to the development effort to require contractors to provide sufficient documentation when they deliver their product. To meet this challenge, a best practice for long-lasting systems includes a contract that delivers adequate architecture documentation to support modernization and sustainment activities.
  
  Adequate architecture documentation enables the sustainment of architecture over decades. If the architecture is provided in the form of a model, that modeling tool should be used to generate an export of the model in a format for use in other tools that might be used by the sustainment organization. Moreover, the sustainment team needs processes in place to permit periodic updates. The team also needs the discipline to execute processes to update the architecture as changes occur.
  
  
• There should be a set of explicit and prioritized set of mission goals and objectives for every project. Without priorities, principled design tradeoff analysis and evaluation cannot be performed. Prioritized goals and objectives should be defined early and then used throughout the development process to ensure modernization activities are in alignment with them. Organizations can fall victim to diverging baselines sometimes within projects, but also with how those project baselines fit into the larger program. Teams must manage configurations that are part of--and have implications for--enterprise capabilities, such as networking, information assurance, and data management. This responsibility requires new perspectives on what to manage at what level (system or enterprise) and with what stakeholder involvement. Competing mission goals and multiple teams making changes at the same time is a recipe for disaster. By updating the systems engineering plan (SEP) to explicitly capture, communicate, and prioritize architectural goals and objectives, the sustainment team can properly perform design tradeoffs.
  
  The organization should use the engineering plan as a living document with frequent updates to reflect project reality. The plan can also be used to communicate the organization's prioritized goals and objectives. An architecture best practice is to prioritize goals and objectives early and then use them throughout the lifecycle to ensure they align with modernization activities. Of course, these prioritized goals should be reevaluated and re-prioritized as times change.
  
  
• Steps should be included to identify whether or not a change is architecturally significant. When an architecturally significant change is made, the architectural impacts of making the change should be reviewed beforehand to ensure that the architecture is still fit for its intended purpose and the impacts to quality attributes are well understood. All decisions and/or assumptions that are made--along with their rationale--should be documented clearly so that future maintainers understand why the architecture is the way it is and how future changes may impact it. The SEP should include steps to identify architecturally significant requirements (e.g., architecture constraints and quality attribute requirements) along with steps to record the architecture impacts and all decisions made along with their rationale. Moreover, traceability among decisions made, requirements, goals, and architecture changes should be captured as well. These areas are topics that apply across a range of customer organizations and can be applied to both large and small organizations. This series of four blogs will examine each of the four areas, and provide a brief description about some of the challenges and recommendations made, requirements, goals, and architecture changes should be captured as well.

The SEI technical report A Structured Approach for Reviewing Architecture Documentation includes a checklist for addressing the impact of changes on an architecture:

These recommendations can be applied to many different types of projects or programs. For more detailed discussion of software architecture, start with the SEI's Architectural Coaching Fact Sheet.

As military software organizations evolve from development to sustainment, they need to address how to best plan for and execute sustainment activities in an efficient, effective manner. We offer three main recommendations in this blog:

• Architecture documentation developed by the contractor should be delivered as part of the contract.
• Every project should have a set of explicit and prioritized set of mission goals and objectives.
• Steps should be included to identify whether a change is architecturally significant or not.

The architecture provides the framework for how the system will work together, particularly the system quality attributes. Sustaining organizations need to ensure they receive and continue to update the system architecture as they undergo modernization changes during the lifecycle of the system.
We will continue our discussion about what acquisition teams need from an engineering sustainment and our series will continue this thread and discuss modernization and sustainment activities such as cybersecurity techniques, Agile tools, and methods and software automation.

Additional References

This blog is being written at the same time as the Defense Innovation Board (DIB) is writing their Software is Never Done: Refactoring the Acquisition Code for Competitive Advantage

Bass, Len; Clements, Paul; Kazman, Rick. Software Architecture in Practice, Third Edition. Addison-Wesley. 2013. ISBN 978-0-321-81573-6

Clements, Paul; Bachmann, Felix; Bass, Len; Garlan, David; Ivers, James; Little, Reed; Merson, Paulo; Nord, Robert; Stafford, Judith. Documenting Software Architectures: Views and Beyond, Second Edition. 2011. Addison-Wesley. ISBN 978-0-321-55268-6.

Nord, Robert L.; Clements, Paul C.; Emery, David; Hillard, Rich. A Structured Approach for Reviewing Architecture Documentation. CMU/SEI-2009-TN-030. 2009.