AI readiness is an established priority for the Department of Defense workforce, including preparation of the workforce to use and integrate data technologies and artificial intelligence capabilities into professional and warfighting practices. One challenge with identifying workers trained in data/AI areas is the lack of formal certifications held by workers. Workers can develop relevant knowledge and skills using non-traditional learning paths, and as a result civilian and federal organizations can overlook qualified candidates. Workers may choose to cultivate expertise on their own time with online resources, personal projects, books, etc., so that they are prepared for open positions even when they lack a degree or other traditional certification.

The SEI’s Artificial Intelligence Division is working to address this challenge. We recently partnered with the Department of the Air Force Chief Data and AI Office (DAF CDAO) to develop a strategy to identify and assess hidden workforce talent for data and AI work roles. The collaboration has had some significant results, including (1) a Data/AI Cyber Workforce Rubric (DACWR) for assessment of skills identified within the DoD Cyberworkforce Framework, (2) prototype assessments that capture a data science pipeline (data processing, model creation, and reporting), and (3) a proof-of-concept platform, SkillsGrowth, for workers to build profiles of their expertise and assessment performance and for managers to identify the data/AI talent they need. We detail below the benefits of these outcomes.

A Data/AI Cyber Workforce Rubric to Increase Usability of the DoD Cyber Workforce Development Framework

The DoD Cyber Workforce Framework (DCWF) defines data and AI work roles and “establishes the DoD’s authoritative lexicon based on the work an individual is performing, not their position titles, occupational series, or designator.” The DCWF provides consistency when defining job positions since different language may be used for the same data and AI academic and industry practices. There are 11 data/AI work roles, and the DCWF covers a wide range of AI disciplines (AI adoption, data analytics, data science, research, ethics, etc.), including the knowledge, skills, abilities, and tasks (KSATs) for each work role. There are 296 unique KSATs across data and AI work roles, and the number of KSATs per work role varies from 40 (data analyst) to 75 (AI test & evaluation specialist), where most KSATs (about 62 percent) appear in a single work role. The KSAT descriptions, however, do not distinguish levels of performance or proficiency.

The data/AI cyber workforce rubric that we created builds on the DCWF, adding levels of proficiency, defining basic, intermediate, advanced, and expert proficiency levels for each KSAT.

Figure 1 illustrates how the rubric defines acceptable performance levels in assessments for one of the KSATs. These proficiency-level definitions support the creation of data/AI work role-related assessments ranging from traditional paper-and-pencil tests to multimodal, simulation-based assessments. The rubric supports the DCWF to provide measurement options of professional practice in these work roles while providing flexibility for future changes in technologies, disciplines, etc. Measurement against the proficiency levels can give workers insight into what they can do to improve their preparation for current and future jobs aligned with specific work roles. The proficiency-level definitions can also help managers evaluate job seekers more consistently. To identify hidden talent, it is important to characterize the state of proficiency of candidates with some reasonable precision.

Addressing Challenges: Confirming What AI Workers Know

Potential challenges emerged as the rubric was developed. Workers need a means to demonstrate the ability to apply their knowledge, regardless of how it was acquired, including through non-traditional learning paths such as online courses and on-the-job skill development. The assessment process and data collection platform that supports the assessment must respect privacy and, indeed, anonymity of candidates – until they are ready to share information regarding their assessed proficiency. The platform should, however, also give managers the ability to locate needed talent based on demonstrated expertise and career interests.

This led to the creation of prototype assessments, using the rubric as their foundation, and a proof-of-concept platform, SkillsGrowth, to provide a vision for future data/AI talent discovery. Each assessment is given online in a learning management system (LMS), and each assessment groups sets of KSATs into at least one competency that reflects daily professional practice. The purpose of the competency groupings is pragmatic, enabling integrated testing of a related collection of KSATs rather than fragmenting the process into individual KSAT testing, which could be less efficient and require more resources. Assessments are intended for basic-to-intermediate level proficiency.

Four Assessments for Data/AI Job Talent Identification

The assessments follow a basic data science pipeline seen in data/AI job positions: data processing, machine learning (ML) modeling and evaluation, and results reporting. These assessments are relevant for job positions aligned with the data analyst, data scientist, or AI/ML specialist work roles. The assessments also show the range of assessment approaches that the DACWR can support. They include the equivalent of a paper-and-pencil test, two work sample tests, and a multimodal, simulation experience for workers who may not be comfortable with traditional testing methods.

In this next section, we outline several of the assessments for data/AI job talent identification:

• The Technical Skills Assessment assesses Python scripting, querying, and data ingestion. It accomplishes this using a work sample test in a virtual sandbox. The test taker must check and edit simulated personnel and equipment data, create a database, and ingest the data into tables with specific requirements. Once the data is ingested, the test taker must validate the database. An automated grader provides feedback (e.g., if a table name is incorrect, if data is not properly formatted for a given column, etc.). As shown in Figure 2 below, the assessment content mirrors real-world tasks that are relevant to the primary work duties of a DAF data analyst or AI specialist.

• The Modeling and Simulation Assessment assesses KSATs related to data analysis, machine learning, and AI implementation. Like the Technical Skills Assessment, it uses a virtual sandbox environment (Figure 3). The main task in the Modeling and Simulation Assessment is to create a predictive maintenance model using simulated maintenance data. Test takers use Python to build and evaluate machine learning models using the scikit-learn library. Test takers may use whatever models they want, but they must achieve specific performance thresholds to receive the highest score. Automatic grading provides feedback upon solution submission. This assessment reflects basic modeling and evaluation that would be performed by workers in data science, AI/ML specialist, and possibly data analyst-aligned job positions.

• The Technical Communication Assessment focuses on reporting results and visualizing data, targeting both technical and non-technical audiences. It is also aligned with data analyst, data scientist, and other related work roles and job positions (Figure 4). There are 25 questions, and these are framed using three question types – multiple choice, statement selection to create a paragraph report, and matching. The question content reflects common data analytic and data science practices like explaining a term or result in a non-technical way, selecting an appropriate way to visualize data, and creating a small story from data and results.

• EnGauge, a multimodal experience, is an alternative approach to the Technical Skills and Technical Communication assessments that provides evaluation in an immersive environment. Test takers are evaluated using realistic tasks in contexts where workers must make decisions about both the technical and interpersonal requirements of the workplace. Workers interact with simulated coworkers in an office environment where they interpret and present data, evaluate results, and present information to coworkers with different expertise (Figure 5). The test taker must help the simulated coworkers with their analytics needs. This assessment approach allows workers to show their expertise in a work context.

A Platform for Showcasing and Identifying Data/AI Job Talent

We developed the SkillsGrowth platform to further assist both workers in showcasing their talent and managers in identifying workers who have necessary skills. SkillsGrowth is a proof-of-concept system, building on open-source software, that provides a vision for how these needs can be met. Workers can build a resume, take assessments to document their proficiencies, and rate their degree of interest in specific skills, competencies, and KSATs. They can search for roles on sites like USAJOBS.

SkillsGrowth is designed to demonstrate tools for tracking the KSAT proficiency levels of workers in real-time and for comparing these KSAT proficiency levels against the KSAT proficiencies required for jobs of interest. SkillsGrowth is also designed to support use cases such as managers searching resumes for specific skills and KSAT proficiencies. Managers can also assess their teams’ data/AI readiness by viewing current KSAT proficiency levels. Workers can also access assessments, which can then be reported on a resume.

In short, we propose to support the DCWF through the Data/AI Cyber Workforce Rubric and its operationalization through the SkillsGrowth platform. Workers can show what they know and confirm what they know through assessments, with the data managed in a way that respects privacy concerns. Managers can find the hidden data/AI talent they need, gauge the data/AI skill level of their teams and more broadly across DoD.

SkillsGrowth thus demonstrates how a practical profiling and evaluative system can be created using the DCWF as a foundation and the CWR as an operationalization strategy. Assessments within the DACWR are based on current professional practices, and operationalized through SkillsGrowth, which is designed to be an accessible, easy-to-use system.

Seeking Mission Partners for Data/AI Job Talent Identification

We are now at a stage of readiness where we are seeking mission partners to iterate, validate, and expand this effort. We would like to work with workers and managers to improve the rubric, assessment prototypes, and the SkillsGrowth platform. There is also opportunity to build out the set of assessments across the data/AI roles as well as to create advanced versions of the current assessment prototypes.

There is much potential to make identifying and developing job candidates more effective and efficient to support AI and mission readiness. If you are interested in our work or partnering with us, please send an email to info@sei.cmu.edu.

Measuring knowledge, skills, ability, and task fulfillment for data/AI work roles is challenging. It is important to remove barriers so that the DoD can find the data/AI talent it needs for its AI readiness goals. This work creates opportunities for evaluating and supporting AI workforce readiness to achieve those goals.