Software Engineering Institute

As the strategic importance of AI increases, so too does the importance of defending those AI systems. To understand AI defense, it is necessary to understand AI offense—that is, counter AI. This paper describes counter AI. First, we describe the technologies that compose AI systems (the AI Stack) and how those systems are built in a machine learning operations (MLOps) lifecycle. Second, we describe three kinds of counter-AI attacks across the AI Stack and five threat models detailing when those attacks occur within the MLOps lifecycle. Finally, based on Software Engineering Institute research and practice in counter AI, we give two recommendations. In the long term, the field should invest in AI engineering research that fosters processes, procedures, and mechanisms that prevent vulnerabilities being introduced into AI systems. In the near term, the field should develop the processes necessary to efficiently respond to and mitigate counter-AI attacks, such as building an AI Security Incident Response Team and extending existing cybersecurity processes like the Computer Security Incident Response Team Services Framework.