Software Engineering Institute

The Department of Defense (DoD) is increasing its use of artificial intelligence (AI) classifiers and predictors; however, users may grow to distrust the results because AI classifiers are subject to a lack of robustness (i.e., the ability to perform accurately in unusual or changing contexts). Drift in data/concept, evolving edge cases, and emerging phenomena undermine the correlations relied upon by AI. New test and evaluation methods are therefore needed for ongoing evaluation. The SEI AIR tool offers a precedent-setting capability to improve the correctness of AI classifications and predictions, increasing confidence in the use of AI in development, testing, and operations decision making.

We are seeking DoD collaborators to use and provide feedback on our technology. As a participant, your AI and subject-matter experts will work with our team to identify known causal relationships and build an initial causal graph. Our process involves using a cutting-edge causal discovery tool, Tetrad, with custom causal identification algorithms and stacked super-learners using doubly robust causal estimators to build an AI “health report.” Your report will include a confidence range of expected treatment effects from your data and interpretations of the causal graph to give you actionable insights into your AI classifier’s health.