Strategy documents from multiple military branches—including the Air Force, Army, and Marine Corps—have emphasized the importance of improved sensing for national security. To improve sensing, the DoD can pursue several approaches. The first is to develop and deploy new kinds of sensing capabilities. This is an essential part of intelligence, surveillance, and reconnaissance (ISR) modernization, but it can be costly and time consuming. A second approach is to leverage existing nonconventional sensors, such as ones available in consumer products or embedded in physical devices. While these sensors may be lower fidelity, they are economical, widespread, and may detect signals in multiple modalities.

In this post, we describe an ISR paradigm called cyber-physical sensing aligned to the second approach described above, where we explore use of existing commercial sensors to enhance the ISR mesh. We begin by summarizing the foundations and promise of the approach. We then discuss recent advances that could enable cyber-physical sensing to achieve significant benefits in the DoD context and beyond. We end by considering open cyber-physical sensing research areas and opportunities to partner with the SEI.

Cyber-Physical Sensing

Cyber-physical systems combine computation, communication, and control capabilities with physical components and processes. For example, healthcare devices that monitor patients’ vitals and transmit signals to providers and homes are equipped with Internet of Things (IoT) devices to automatically manage temperature, lighting, and security. These systems bridge the gap between the digital and physical worlds, enabling the monitoring and control of physical entities through interconnected computer systems.

Cyber-physical sensing uses cyber-physical systems to observe the physical world, ideally in flexible, scalable, and opportunistic ways. Cyber-physical sensing includes three core concepts:

• Multi-sensor exploitation—The number of network-connected IoT devices has risen from 9.7 billion in 2020 to 15.3 billion in 2023 and is expected to reach 29.4 billion by 2030. IoT devices are ubiquitous, spanning various domains such visual, audio, electromagnetic (EM), magnetic, chemical, and mechanical sensing. Rather than building new platforms, cyber-physical sensing seeks to harness existing multi-sensor networks.
• Multi-intelligence data processing—The volume of data generated by opportunistic cyber-physical sensing networks can be very large, and the data may be noisier than signals arising from purpose-built high-fidelity ISR sensors. Multi-intelligence data processing is the application of algorithms, including AI, to extract reliable information from data captured by cyber-physical sensing networks.
• Enhancing warfighter capabilities—Cyber-physical sensing systems can gather data in degraded and denied sensing environments, detect overlooked emissions, create redundant networks comprised of low-cost sensors, and accelerate decision-making by using sensors and compute placed at the tactical edge. These attributes of cyber-physical sensing not only enhance existing warfighter capabilities, but they also pave the way for entirely new ones.

The Rise of Cyber-Physical Sensing

Cyber-physical sensing is not a new concept. However, recent advances in three areas have positioned cyber-physical sensing to fulfill its transformational potential. As shown in Figure 1, these areas are sensors, algorithms, and computing resources.

Sensors are devices designed to capture signals from the physical world. Key advances in this area include the expanding capability, diversity, and accessibility of consumer-grade sensing devices, coupled with their low costs. For example, smartwatches, fitness rings, wrist bands, and other fitness trackers, which were not available 10 years ago, have now become ubiquitous. These devices come equipped with various sensors, such as heart rate monitors, accelerometers, gyroscopes, and GPS. Beyond the hardware aspect, the proliferation of large-scale datasets derived from sensors has accelerated the evolution of effective methods for processing multi-modal signals.

Algorithms refer to the computer programs for extracting information from raw signals. The main driver in this area stems from deep learning, a subarea of machine learning (ML) that involves training neural networks to extract complex patterns and features from raw data. Deep neural networks (DNNs) can process many different types of inputs including images, video, audio, and more. DNNs require a large amount of data and are time-consuming to train. Recent progress in transfer learning has mitigated these challenges by allowing pre-trained networks—those previously trained for similar tasks—to adapt to new contexts and domains with reduced training needs.

Computing resources refer to the hardware for implementing cyber-physical sensing signal processing. Specialized hardware like graphics processing units (GPUs) and tensor processing units (TPUs) allowed DNNs to be applied in feasible time in the first place. The movement toward low-powered, rugged devices enables ML algorithms to be applied toward the edge of operational networks. Additionally, in networked environments, cloud computing resources allow algorithms to process data in near real-time and at scale.

Cyber-Physical Sensing in Action

The following vignette focused on defensive counterair illustrates the core cyber-physical sensing concepts in a DoD context.

By integrating these existing sensors, the base can form an ad hoc sUAS detection and tracking system through cyber-physical sensing. In addition to tracking the sUAS, this network can also detect other objects or events of interest, such as a detonation on the runway or an unauthorized individual approaching the base perimeter.

Combining Cyber-Physical Sensing with Dedicated Sensors and Platforms

Cyber-physical sensing is not intended to replace existing and planned ISR capabilities. However, it can complement these capabilities in some of the following ways:

• Resiliency—As of 2023, the United States Air Force included about 504 aircraft for ISR, battle management, and command and control in its total active inventory. Across services, including the Space Force, the number of ISR platforms is far greater. Nevertheless, there are vastly more cyber-physical sensing devices than dedicated sensing assets, and these devices can be readily acquired. Given the potential scale of cyber-physical sensing networks, it becomes impossible to destroy enough sensors to meaningfully degrade performance.
• Cost—The cost to acquire dedicated ISR platforms ranges from tens to hundreds of millions of dollars. In contrast, the cost of an iPhone 15, a highly capable, multi-sensor device, is less than $1,000. Given the availability of cyber-physical sensing devices, they can be fielded at scale, and they cost more to attack than to produce.
• Coverage—Given the limited number of air- and space-based platforms, it is not possible to perform 24/7 monitoring of all sites. Cyber-physical sensing, which leverages sensors in place, may open new avenues for persistent monitoring.
• Diversity—Existing ISR capabilities mainly use electro-optical (EO), infrared, synthetic aperture radar (SAR), electronic intelligence (ELINT) and signals intelligence (SIGINT) sensors. However, objects emit in other domains. Cyber-physical sensing devices may detect audio, mechanical, chemical, magnetic, electrical, and other signals overlooked by existing sensors.
• Speed—By using portable sensors and computing devices, cyber-physical sensing may enable signal processing on the battlefield by warfighters. This accelerates delivery of information for tactical decision making.

In the envisioned ISR future, cyber-physical sensing does not replace dedicated ISR platforms. Rather, cyber-physical sensing fills the temporal, spatial, and domain gaps that exist within the existing ISR mesh. Further, cyber-physical sensing serves as a first line sensing grid used to cue more capable, low-density sensors from the ISR inventory.

Challenges for Cyber-Physical Sensing

Based on our work in cyber-physical sensing, we have identified significant remaining challenges. Table 1 summarizes these open research areas.

One set of challenges involves establishing robust cyber-physical sensing networks. The use of various protocols, data formats, and standards among different devices can hinder interoperability, particularly when networks are formed post hoc rather than ad hoc. Moreover, as the number and diversity of sensing devices increases, managing and scaling the network becomes more difficult. Additionally, some cyber-physical sensing networks are intended for operation in contested and degraded environments, necessitating robust performance in the face of potential failures.

Another set of challenges involves developing effective algorithms for cyber-physical sensing. While substantial research focuses on classifying visual and audio signals, comparatively less attention is directed towards classifying signals from other domains, such as mechanical or electromagnetic signals, or conducting multi-modal classification. Furthermore, the availability of high-quality datasets for training and evaluating cyber-physical sensing models in other domains is limited.

The final set of challenges involves deploying cyber-physical sensing systems and networks. Algorithms must be deployable under operational conditions, including scenarios involving the tactical edge and low-powered devices. This requirement underscores the need for algorithms that can run efficiently on portable computing devices. Additionally, cyber-physical sensing networks introduce a complex attack surface that necessitates security measures to safeguard against potential threats.

Partner with Us

Cyber-physical sensing offers a flexible, cost-effective, and timely paradigm to extend the national ISR mesh. A growing collection of examples illustrate the advantages that may be gained from cyber-physical sensing. Nevertheless, a formidable set of challenges remain for cyber-physical sensing to achieve its full potential. Given these challenges, how can organizations harness cyber-physical sensing effectively and safely?

Here at the SEI, we have a long history of work on cyber-physical systems and expertise across our AI, Software Solutions, and CERT Divisions to create cyber-physical systems that are effective, practical, and secure. As you explore opportunities to develop more capable cyber-physical sensing capabilities, or to leverage existing ones, we are ready to share our experiences and lessons learned.