A Retrospective in Engineering Large Language Models for National Security
• White Paper
Software Engineering Institute
At the request of the White House, the Office of the Director of National Intelligence (ODNI) began exploring use cases for large language models (LLMs) within the Intelligence Community (IC). As part of this effort, ODNI sponsored the Mayflower Project at Carnegie Mellon University’s Software Engineering Institute from May 2023 through September 2023. The Mayflower Project attempted to answer the following questions:
- How might the IC set up a baseline, stand-alone LLM?
- How might the IC customize LLMs for specific intelligence use cases?
- How might the IC evaluate the trustworthiness of LLMs across use cases?
This document discusses the findings and recommendations from the Mayflower Project and provides additional background information about LLMs and how they can be engineered for national security use cases. This report also describes lessons learned at several stages of the engineering process: building a baseline LLM, tuning an LLM for national security use cases, and evaluating LLMs for trustworthiness.