This post is also authored by Angela Horneman and Andrew Mellinger.

Artificial intelligence (AI) is driving advances throughout modern society, including in the fields of medicine, transportation, education, and finance. In the government space, the Department of Defense (DoD) has made the advancement of AI a priority "to maintain its strategic position to prevail on future battlefields and safeguard a free and open international order."

In this landscape, AI systems are being developed at a rapid-fire pace, yet the discipline of AI engineering is still relatively new and evolving. The very nature of AI systems makes foundational practices that are critical to the field of software engineering a massive challenge. In this blog post, we present 11 AI engineering practices, excerpted from a recently published paper, that we have developed to bring order to this landscape and lay the groundwork for establishing AI engineering as a discipline.

A Growing Demand for AI Solutions

Our AI engineering practices represent an important first step in helping the DoD meet a growing demand for AI solutions by highlighting the importance of awareness, planning, and communication. In September, the Air Force released its 2019 artificial intelligence strategy. "Al is a capability that will underpin our ability to compete, deter and win... It is crucial to fielding tomorrow's Air Force faster and smarter, executing multi-domain operations in the high-end fight, confronting threats below the level of open conflict and partnering with our allies around the globe," Acting Secretary of the Air Force Matthew Donovan and Air Force Chief of Staff General David Goldfein wrote in an Air Force news release announcing the strategy.

We designed these practices to help decision makers balance short-term and long-term needs of an AI engineering project. They guide decision makers in forming a team and project plan that will create a successful system that fulfills needs while being reliable, maintainable, and secure. Without these practices, planning does not account for the full complexity of the systems, maintenance needs come as a surprise, and end-user concerns are unexpected.

Perspectives on AI

In considering foundational practices for AI engineering, we relied, in part, on our experience, general subject matter expertise, and a variety of published works from industry leaders with experience in developing AI-enabled software systems. Several of the lessons discussed in this literature overlapped with our own observations, providing early validation of our efforts.

More significantly, we focused on the same rigor that we bring to the discipline of software engineering. As we noted in our paper outlining these practices, an AI system is, in fact, a software-intensive system. We focus on four qualities of software-intensive systems that are also essential to building, operating, and defending AI systems:

• Software continually enables new capabilities that deliver the goals for the system. The SEI has long been at the leading-edge of research and development in software architecture, which is the primary carrier of system qualities such as performance, modifiability, and long-term viability.
• With enhanced and new capabilities comes the need, particularly pointed in AI systems, that software be secure and trustworthy, two related but still separate aspects. Software in any modern system is at risk of malicious tampering before it is integrated into an operational system and once the system is in operation¾a circumstance that is exacerbated in an AI system that will continually learn and evolve. Flaws in software that can be exploited or that go undetected will make the AI system vulnerable: The SEI is pioneering tools and practices for the automated discovery and repair of vulnerabilities. Likewise, the AI system has to be trustworthy, since it produces probabilistic outcomes. Thus, there is human-machine interaction, which must become human-machine teaming.
• Further, even more than with software-intensive systems, organizations will need to go beyond rapid deployment with AI systems to incorporate DevOps, to accommodate emerging behavior from adaptation in use.
• Finally, the AI system has to be affordable. The SEI seeks to give organizations the means to predict and control software cost. Current SEI research for that goal applies causal modeling algorithms and tools to a large volume of project data to identify, measure, and test causality.

11 Foundational Practices

It is important to note that when engineering an AI system, these practices are all initial entry points, and many rely on each other. For example, Practice 1 will need to derive from requirements and data science while Practice 9 relies heavily on existing design and architecture know-how as well as emerging technologies and design approaches that will guide these systems. Other practices listed below are more advisory, such as Practice 10, which advocates allocating sufficient "computing, hardware, storage, bandwidth, expertise, and time."

Here are the 11 foundational practices of AI engineering, which are geared toward decision makers. The recently published white paper, AI Engineering: 11 Foundational Practices, which we coauthored, provides a full description of each practice:

1. Ensure you have a problem that both can and should be solved by AI, so that you know both the outcomes you want to achieve and the data you will need to achieve them. Start with a well-defined problem, understanding what you want to accomplish and the outcomes you need, while ensuring you have data available to infer those outcomes.
2. Include highly integrated subject matter experts, data scientists, and data architects in your software engineering teams. AI engineering teams consist of experts in the problem domain (subject matter experts), data engineering, model selection and refinement, hardware infrastructure, and software architecting, in addition to the other typical software engineering expertise.
3. Take your data seriously to prevent it from consuming your project. Data ingestion, cleansing, protection, monitoring, and validation are necessary for engineering a successful AI system--and they require tremendous amounts of resources, time, and attention.
4. Choose algorithms based on what you need your model to do, not on their popularity. Algorithms differ in several important dimensions: what kinds of problems they can solve, how detailed the information in the output is, how interpretable the output and models are, and how robust the algorithm is to adversaries (via manipulating training data, interfering with a feedback loop, and the like).
5. Secure AI systems by applying highly integrated monitoring and mitigation strategies. The attack surface of an AI system is expanded due to challenges with understanding how its complex models function and depend on data. These additional attack surface dimensions compound the vulnerability of the traditional hardware and software attack surface.
6. Define checkpoints to account for the potential needs of recovery, traceability, and decision justification. AI systems are acutely sensitive to the dependencies among input data, training data, and models. Changes to the version or characteristics of any one can quickly--and sometimes subtly--affect others.
7. Incorporate user experience and interaction to constantly validate and evolve models and architecture. As much as possible, use an automated approach to capture human feedback on system output and improve (i.e., retrain) models. Monitor user experience to detect issues early, such as degraded performance in the form of system latency or reduced accuracy.
8. Design for the interpretation of the inherent ambiguity in the output. AI output requires much more interpretation than that of most other systems. The uncertainty introduced by an AI system might not be acceptable under certain scenarios for the mission and users.
9. Implement loosely coupled solutions that can be extended or replaced to adapt to ruthless and inevitable data and model changes and algorithm innovations. The boundaries between the components of an AI system deteriorate more quickly than those in traditional systems due to the entanglement of data. Moreover, the impact of change is heightened due to unanticipated direct and indirect data dependencies.
10. Commit sufficient time and expertise for constant and enduring change over the life of the system. Teams significantly underestimate resources needed nine out of ten times. Building AI systems requires greater resources initially that need to scale up quickly and significant dedication or resources through the life of the system.
11. Treat ethics as both a software design consideration and a policy concern. Evaluate every aspect of the system for potential ethical issues. Account for organizational and societal values in all aspects of the system, from data collection, to decision making, to validation and monitoring of performance and effectiveness.

Looking Ahead

While these 11 practices provide some initial guidelines for DoD executives, program managers, and technical leadership, they will likely evolve and advance along with the field of AI engineering.

As we begin the national conversation on AI engineering, these recommendations provide some initial guidelines to help DoD executives, program managers, and technical leadership ask some of the right questions in the early stages of building and adopting AI technologies and capabilities. At the SEI we have several research projects focused on advancing AI, including several that help organizations with rapid prototyping and building models and data pipelines. Those research projects will be discussed in this space in the coming months.

In the near term, we are forming a community of interest around AI engineering for defense and national security. As part of this community of interest, collaborators, customers, stakeholders, and subject matter experts will work together to understand and set the course for AI engineering. If you are interested in being part of the community of interest, contact us to get involved.

Additional Resources

Download the paper AI Engineering: 11 Foundational Practices.