A Retrospective in Engineering Large Language Models for National Security

September 29, 2023 • White Paper

By

Shannon Gallagher, Andrew O. Mellinger, Jasmine Ratchford, Nick Winski, Tyler Brooks, Eric Heim, Nathan M. VanHoudnos, Swati Rallapalli, William Nichols, Bryan Brown, Angelique McDowell, and Hollen Barmer

This document discusses the findings, recommendations, and lessons learned from engineering a large language model for national security use cases.

Publisher

Software Engineering Institute

Abstract

At the request of the White House, the Office of the Director of National Intelligence (ODNI) began exploring use cases for large language models (LLMs) within the Intelligence Community (IC). As part of this effort, ODNI sponsored the Mayflower Project at Carnegie Mellon University’s Software Engineering Institute from May 2023 through September 2023. The Mayflower Project attempted to answer the following questions:

How might the IC set up a baseline, stand-alone LLM?
How might the IC customize LLMs for specific intelligence use cases?
How might the IC evaluate the trustworthiness of LLMs across use cases?

This document discusses the findings and recommendations from the Mayflower Project and provides additional background information about LLMs and how they can be engineered for national security use cases. This report also describes lessons learned at several stages of the engineering process: building a baseline LLM, tuning an LLM for national security use cases, and evaluating LLMs for trustworthiness.

Software Engineering Institute