Delivering Resilient Software Capability to the Warfighter at the Speed of Relevance

In March, the secretary of defense directed the Department of Defense (DoD) to adopt the Software Acquisition Pathway (SWP) to accelerate the development and deployment of capabilities to the warfighter. The directive to default to the SWP arrives at a time when DoD missions increasingly rely on software and the related technologies of cybersecurity and artificial intelligence (AI)—all of which are focus areas at the SEI. These technical areas grow in importance as national security and defense organizations require enhanced capabilities to protect a broader range of targets against more sophisticated and adept threats.

In this post, I will highlight the ways in which our research and development support DoD’s use of modern software practices at each phase of the software development and operation lifecycle.

The SEI and the Software Acquisition Pathway

The SEI’s depth of experience with data-driven techniques, methods, and approaches; software engineering; and acquisition science catalyzed our work on the SWP.

Section 255 of the FY2020 National Defense Authorization Act (NDAA) called on the DoD to orient its software activities in research, development, testing, and acquisition toward modern software engineering practices described in two studies that SEI also contributed to: the Defense Innovation Board’s 2019 Software Acquisition and Practices (SWAP) and the Defense Science Board Task Force’s 2018 Design and Acquisition of Software for Defense Systems.

As part of our SWP work, the SEI worked hand-in-hand with stakeholders across the DoD and the defense industrial base in the iterative development, testing, and updating of the policy and supporting tools and resources. As called for in the policy document DoDI 5000.87, DoD programs are to emphasize risk-based cybersecurity throughout the lifecycle, relying on modern software practices including DevSecOps. Cybersecurity and DevSecOps form a foundation on which these programs can incorporate new AI technologies more securely.

Since the SEI’s inception as a federally funded research and development center (FFRDC), we have worked with academia, government, and industry to conduct research and help DoD programs apply these essential tools, techniques, practices, and policies. The SEI prioritizes development of techniques to support continuous, resilient, and timely deployment of software capability for the warfighter while ensuring that software system performance and security are not compromised in mission-critical settings.

SEI Impact on Modernizing DoD Software Development

Over the last decade, the DoD has been incorporating principles and practices that promote continuous, iterative deployment of software capability. One such initiative was the development of the 250-plus member Agile Collaboration Group. Members share lessons learned to help DoD practitioners more readily benefit from using Agile methods in their larger-scale systems. When the group began in 2012, it focused on overcoming adoption challenges around Agile. In 2017, it expanded into DevSecOps research, development, and field engagement.

In 2024 the SEI conducted a study examining the state of DevSecOps in the DoD, the results of which were recently released by the DoD Chief Information Officer (CIO). The study found that while certain programs have had success adopting DevSecOps practices, the DoD still needs to implement these successes at scale. The study holds in part that

• Investing in DoD software factories is key to securing our future capability.
• DevSecOps is a key strategy to accelerate delivery time.
• Success rests on reimagining a mission-ready DevSecOps workforce.
• Strong leadership committed to creatively driving solutions is critical to overcoming barriers.

The DevSecOps study also recognizes that it is necessary to align those practices with the mission.

To a significant degree, the effective use of modern software practices rests on recognizing the strategic value of data that is now available to the DoD in increasingly massive amounts. To give DoD analysts greater visibility into DevSecOps pipeline data, the SEI recently released Polar, a solution to the limitations of traditional batch data processing. Polar gives visibility into the current state of an organization’s DevSecOps infrastructure, allowing for the entirety of the data to be engaged for informed decision making. The Polar framework, which can be downloaded from the SEI’s GitHub site, helps DevSecOps organizations monitor and gain insights into security aspects and address the challenges posed by building complex software systems in highly regulated environments.

DoD program leaders must increasingly manage all of the software development capability. This responsibility often means dealing effectively with the technical debt that can accumulate in aging systems as well as that which can accumulate in rapid, iterative development. The SEI has been a pioneer in developing and applying technical debt management practices in complex systems for national security and defense. Beginning in 2010, the SEI challenged the software engineering research community to find ways to manage technical debt and convened annual workshops on the topic. Those workshops produced case studies, empirical results from applying techniques, and comparisons of tools shared by the SEI and the software community in hundreds of publications in the Association for Computing Machinery (ACM) and IEEE digital libraries. In 2018, the SEI’s community efforts resulted in the first international conference on the subject, TechDebt. The eighth TechDebt conference was held in April 2025.

The SEI also led through ground-breaking research on the topic. An early paper on the topic, In Search of a Metric for Managing Architectural Debt, authored by SEI and University of British Columbia researchers, received the Most Influential Paper Award for its lasting impact on software architecture research and practice at the 2022 IEEE International Conference on Software Architecture. The SEI’s expertise in technical debt R&D is the reason that the DoD commissioned a team of our software development experts to write a report addressing the NDAA 2022 Section 835 mandate, delivered to Congress in December 2023.

The SEI has used its role to enable the DoD’s effective use of modern software engineering practices in other ways, as well, including tools to

• support the development of secure coding standards for C, C++, Java, Perl, and the Android platform (since 2005)
• help DoD program managers analyze software value versus cost (since 2015)
• automate the identification and, often, the correction of software flaws (since 2017)
• model and reduce software risk (since 2018)
• implement efficient software factories (since 2019)
• automate large-scale refactoring (since 2019). See our website for more information on software evolution and isolation.

SEI Impact on Modernizing DoD Software in Operation

A core tenet of the SEI’s mission is to enable the DoD to rapidly deploy resilient software capabilities. To do this, they need the right tooling to use modern software practices and the means to assure system performance.

To facilitate DevSecOps use with large-scale systems, the SEI created the Platform-Independent Model (PIM)—available on the SEI’s GitHub site—to describe a DevSecOps pipeline at the highest level: requirements, the product development lifecycle process, and the organizational roles needed to produce software. Since its release, a cross-disciplinary SEI team has enhanced the PIM by incorporating threat scenarios: attack type, actors, effects, and pipeline assets for protection. The resulting upgraded tools can be used to create more secure processes and pipelines or spot security weaknesses in existing ones. When the pipeline is more secure, so too is the software it produces.

To improve software risk analysis, SEI researchers and tool developers recently released an open-source tool that streamlines and automates quality assurance testing and analysis, Silent Sentinel. This tool provides a repeatable, consistent process to give system stakeholders a realistic assessment of how an application will affect their deployment environment.

Continuous delivery of software capability also means that systems using those capabilities need continuous assurance of safety, security, and other qualities. In ongoing work, the SEI is seeking to reduce the time and effort required to re-assure large systems. This notion of system assurance extends beyond security to encompass multiple architecturally significant concerns, including performance, modifiability, safety, and reliability.

SEI Impact on AI System Assurance

The world, and certainly the DoD’s technology environment, is becoming increasingly AI-augmented. When AI systems for national security fail in development or operation, they cause serious, real-world consequences. Unfortunately, there are few accepted best practices for testing AI systems due to the challenges of properly defining requirements and evaluating criteria.

In 2023, the Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E)) and the SEI launched a center aimed at establishing methods for assuring trustworthiness in AI systems with emphasis on interaction between humans and autonomous systems. The Center for Calibrated Trust Measurement and Evaluation (CaTE) aims to help the DoD ensure that AI systems are safe, reliable, and trustworthy before being fielded to operators in critical situations.

Further, as detailed in a recent blog post and podcast, a group of SEI software and AI experts recently introduced Machine Learning Test and Evaluation (MLTE), a new process and tool jointly developed by the SEI and the Army AI Integration Center (AI2C) to create safer, more reliable ML systems. MLTE addresses three problems common in the ML model development process that are barriers to effective test and evaluation processes.

1. Communication barriers between product development team members. Team members are often siloed across organizations, leading to problems in gathering ML model requirements cognizant of the system context and communicating ML model evaluation results.
2. Documentation problems for ML model requirements. Eliciting and documenting ML model requirements is often a challenge for organizations, and documentation for ML system requirements is often missing or of low quality.
3. Requirement evaluation. Even if requirements are properly defined and documented, there is no ML-specific method to support their implementation, testing, and evaluation.

Broadening our perspective to AI risk management, we explored how to conceptualize modern AI risk management frameworks (RMFs) analogous to those for cyber risk. This work illustrates the broad scope of challenges that AI Engineering practices must address, including software engineering and cybersecurity considerations. A recent SEI blog post noted this:

We must consider, in other words, the behavior of a system or an associated workflow under both expected and unexpected inputs, where those inputs may be particularly problematic for the system. It is challenging, however, even to frame the question of how to specify behaviors for expected inputs that are not exactly matched in the training set. A human observer may have an intuitive notion of similarity of new inputs with training inputs, but there is no assurance that this aligns with the actual featuring—the salient parameter values—internal to a trained neural network.

SEI research teams also saw the need for an AI security response team analogous to computer security response. An informed and motivated attacker may deliberately manipulate operational inputs, training data, and other aspects of the system development process to create circumstances that impair correct operation of an AI system. To address this need, the SEI introduced the first-of-its kind AI Security Incident Response Team (AISIRT).

To assure that future AI systems will be robust, secure, scalable, and capable of serving warfighter needs, the SEI has been leading the initiative to advance the discipline of AI Engineering. This emergent discipline will enable practitioners to focus R&D efforts in AI on developing tools, systems, and processes for national security contexts.

SEI Impact on Software Acquisition Security

Virtually all products and services that a DoD program acquires are supported by or integrate with information technology that includes third-party components or services. Practices critical to monitoring and managing these risks can be scattered, resulting in inconsistencies, gaps, and slow response to disruptions. To address these issues, SEI researchers created the Acquisition Security Framework (ASF), which provides the DoD with a roadmap for building security and resilience into a system rather than bolting them on after deployment. The ASF promotes better communication and information sharing across all program and supplier teams to coordinate their management of engineering and supply chain risks. In this way, the ASF helps programs match threats in a dynamic environment with the rapid evolution of needed software capabilities.

In addition, in early 2020 the SEI partnered with Johns Hopkins University Applied Physics Laboratory (APL), a university affiliated research center, to release the initial version of the cybersecurity maturity model at the heart of the Cybersecurity Maturity Model Certification (CMMC) program. CMMC provides the DoD Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD(A&S)) with a powerful tool to improve risk-informed decisions and contractor security in the defense industrial base supply chain.

SEI Impact on the Future of Software Engineering Research

An important part of the SEI mission is to anticipate both challenges and opportunities in its three mission areas: software engineering, cybersecurity, and AI. To develop an agenda for the next decade of software engineering research, the SEI brought together an advisory board of visionaries and senior thought leaders to develop an agenda for the next decade of software engineering research. This effort led to the 2021 publication, Architecting the Future of Software Engineering: A National Agenda for Software Engineering Research and Development. The study is a catalyst for research and development at Carnegie Mellon University and the SEI in areas such as AI-augmented software development, the assurance of continuously evolving software systems, and engineering AI-enabled software systems.

That SEI-led study is influencing the DoD software ecosystem and provoking follow-on activities. For instance, in 2023 the SEI and the Networking and Information Technology Research and Development (NITRD) program organized and hosted the U.S. Leadership in Software Engineering and AI Engineering workshop; see the workshop’s executive summary. In addition, we worked with the National Defense Industrial Association’s Emerging Technologies Institute (NDIA ETI) to offer recommendations for the DoD as it shapes its software modernization activities and research portfolio. Further, the SEI is partnering with the software engineering and AI communities to implement the recommendations of the research agenda. The SEI partnered with the Federal Aviation Administration and Vanderbilt University to convene two workshops in 2024 to address the assurance of continuously evolving software systems, one of the study’s areas of focus. SEI researchers along with leaders from the software engineering community will be leading a workshop to address how AI might transform end-to-end software development workflows in the 2025 International Workshop on Envisioning the AI-Augmented Software Development Lifecycle, collocated with the ACM International Conference on the Foundations on Software Engineering.

The SEI: Persistent Focus on Software Modernization

Through the SEI Blog and our podcast series, we highlight the work of our researchers to help the DoD make software a strategic advantage through integrating our domain expertise in AI, cybersecurity, and software. To help the DoD deliver resilient software capability at the speed of relevance, the SEI researches complex engineering problems; creates, prototypes, and refines innovative technologies; and transitions maturing solutions into practice to promote DoD mission success.