This blog post is also authored by Josh Hammerstein.

There are many opportunities for front-line soldiers to use cyber tactics to help them achieve their missions. For example, a soldier on a reconnaissance mission who enters a potentially hostile or dangerous space, such as a storefront in enemy territory, might be able to gain access to an open wireless access point in the area or exploit vulnerabilities in the building's alarm-communication system. These exploits would allow the soldier to provide indicators and warnings to other soldiers in the area about possible enemy activity and threats. Soldiers can expand their arsenal through greater awareness of specific lethal and non-lethal cyber tactics available to them. This blog post describes a new prototype tool developed at the SEI designed to help the soldier identify and exploit cyber opportunities in the physical environment.

"Affordances" are opportunities presented by physical objects "at the interface between...individual(s) and the world." They are potential cyber actions that soldiers could take in their immediate geographical surroundings. The SEI Cyber Affordance Visualization in Augmented Reality (CAVIAR) prototype demonstrates how augmented reality--"a live direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data"--could be used by soldiers to visualize cyber affordances.

CAVIAR works through a head-worn device that displays virtual content to the soldier in the form of 3D graphics that are aligned and overlaid on the real world. The information identifies specific actions that the soldier could take related to objects in the soldier's field of vision, such asdownload electronic information from a computer, exploit a network vulnerability in a router, disable an alarm system, etc.

CAVIAR is intended to help front-line soldiers visualize cyber-warfare opportunities in their natural surroundings. Here is a video that demonstrates how CAVIAR works.

Currently, the infantry exploits cyber opportunities by having a cyber expert in a nearby vehicle or at a standoff distance communicating over a radio with the soldier in real time or prior to the mission. Such an expert might say to the soldier, "In this house, we think there's an opportunity to gain access to a laptop with useful information." CAVIAR integrates this kind of expert advice into the direct real-time experience of the soldier in the field. Using Microsoft HoloLens, a technology developed for the Xbox gaming system, CAVIAR presents visualizations of cyber affordances in the physical environment. If soldiers wearing the CAVIAR device were to encounter a laptop in their field of vision, the system would recognize the laptop, query a backend database for potential actions that could be taken on that laptop, and then display to the soldier suggestions such as, "Put in your USB drive, get a memory image, and bring it back."

CAVIAR is a prototype that demonstrates how such a tool could improve the situational awareness and increase the knowledge of the soldier in the field. In the example of the house shown in the video above, a tool such as CAVIAR would allow a solider, if permitted by their rules of engagement, to visualize opportunities to employ cyber effects in the surrounding area. These cyber effects include scanning the local network in the house to look for documents, services, and other sources of intelligence. This intelligence could provide soldiers the ability to accomplish a mission without requiring interaction with the local population, thereby reducing risk to the soldier and providing fewer clues to potential adversaries about mission goals.

Augmented reality presents virtual content to the user--in this case, synthetic 3D graphics. It registers and aligns that content with the real world. Graphics are aligned and displayed in conjunction with the objects in view in the real-world setting of the room. This alignment is interactive--it changes as the user moves--and it takes place in real time. A well-known example of partial augmented-reality capability is the yellow first-down line shown on a TV broadcast of a football game. The yellow line aligns with the view of the field and changes in conjunction with the view on the TV screen. However, because the rendering is not done in real time and is not interactive, this example is not pure augmented reality. As outlined in a recent Economist article, augmented-reality technology has improved and now provides the ability to track movement and build a map based on the natural features in the field of vision, such as corners and hard edges of desks and tables. Microsoft HoloLens provides this localization tracking.

While using the commercially available Microsoft HoloLens would help to make the CAVIAR device affordable for the government, the challenge for the military, and for the SEI in developing CAVIAR, is to make it rugged and survivable in situations for which it was not originally designed. For CAVIAR, the SEI has written software for the HoloLens display that allows a server computer to tailor the view displayed for combat situations on the client computer, which is the device worn by the soldier. If used in a real-world setting, someone at the server end would use the SEI software to program what the end user sees. A database, possibly mounted on a ground or air vehicle, would be accessible on the battlefield ad hoc network and loaded with synthetic icons or graphics for use in the display seen by the soldier. These visuals would then be overlaid onto the view of the real world provided by the client computer for instructions, navigation, and guidance.

Despite their potential, cyber affordances have been leveraged more by intelligence agencies than by the DoD in combat situations. Former Army Cyber Commander General Edward Cardon articulated a vision, Cyber Support to Corps and Below, for making cyber capabilities accessible to the average front-line soldier, and it was this vision that motivated the SEI's development of CAVIAR. The current name of this strategy within the U.S. Army is Expeditionary Cyber Operations (Army FM 3-12).

In addition to exploitation of offensive opportunities, a tool such as the CAVIAR tool would raise awareness of affordances that an enemy could exploit, thereby raising awareness of the vulnerabilities of cyber-connected objects and improving defensive capabilities. The system would also provide dual-use possibilities. For example, it could take pictures of the field of view for later use, or provide facial recognition of individuals and access to information about them, such as a Facebook page.

Getting end users to adopt augmented-reality applications such as CAVIAR presents one risk to the SEI's vision. First developed by Boeing for use on the 747, augmented reality has been in use since the 1990s. In that case, users deemed the devices to be bulky, leading to resistance and rejection. To mitigate such usability risks, the SEI is working with soldiers at Fort Gordon on the CAVIAR prototype. With the help of Professor Steve Fiener and other researchers at Columbia University's Computer Graphics and User Interfaces Lab, the SEI is building a better prototype with improved GPS capabilities.

Wrapping Up and Looking Ahead

The next step for the SEI in developing CAVIAR will be to test the improved prototype in a user study at West Point, the home of the Army Cyber Institute, the Army's think tank for cyber. The Cyber Institute work is led by Lt. Col. Brett Lindberg, a research scientist who is collaborating with the SEI on CAVIAR. At a West Point urban training environment, the unit from Fort Gordon will conduct a mission using the CAVIAR system. The SEI plans to collect data on the mission and will publish results in a forthcoming IEEE paper.

When ready for real-world adoption, a system such as CAVIAR promises to increase knowledge and situational awareness by giving front-line soldiers a more expansive and complete view of the opportunities that cyber tactics afford for achieving their missions.

Additional Resources

View a video that demonstrates how CAVIAR works.