Commercial augmented reality technologies can enhance training, education and operational performance, but they are rarely used by the defence forces. The UK Ministry of Defence is looking at ways to make the most of this technology so that it can become an asset
Augmented Reality (AR) is a live, direct or indirect experience of a physical real-world environment that is modified by supplementary information. This effectively enhances a user’s perception and understanding of their surroundings. In contrast to virtual reality, AR is not simulation, but involves augmentation of the real world, and the technology covers a range of senses including visual, auditory and haptic.
AR hardware and software applications are now increasingly available within the consumer market, such as adding an overlay to televised swimming races to show the world record pace, or Google Glass displays providing supplementary information about the surroundings. However, the technology has yet to be widely exploited by the defence industry.
UK Defence AR research
The UK Ministry of Defence (MoD) is looking at ways to reap greater benefits from AR technology in the defence environment. SEA, part of Cohort PLC and a specialist provider of augmented reality research, has been working on the MoD’s Joint Focus Experimentation 3(JFX3) project, which aims to increase MoD’s understanding of AR technologies, as well as reducing the barriers to their exploitation and use.
JFX3 is part of the Synthetic Environment (SE) Tower of Excellence, a UK MoD research initiative, established to provide a mechanism to enable the ministry, industry and academia to work together on future areas of research for mutual benefit.
The JFX3 project was given the following statement of purpose: “To identify, demonstrate and assess the role of Customer of the Shelf (COTS) and Government of the Shelf (GOTS) AR technologies for defence purposes. To conduct a benefits-focussed evaluation of AR solutions which could be deployed rapidly at a low-cost into defence operations and training, exploiting recent developments in the commercial domain.”
The JFX3 project was divided into two phases. The key findings from Phase 1 were: AR technology is evolving very quickly (faster than other similar technologies) and becoming more accessible via the use of COTS applications on portable devices (e.g. AR browsers available on smartphones and tablets); issues with information criteria (e.g. priority, dynamism, timeliness, format and commonality) and system requirements (collection, processing, network/bearer and display) will need to be considered for any new AR technology and; there are more concerns over barriers to AR within defence, rather than cocerns with the technology itself.
Phase 2 of the project, conducted in 2013, focussed upon the planning, execution and analysis of a field evaluation of AR technologies. In addition to these activities, stakeholder feedback in Phase 1 also prompted further investigation of the barriers to AR adoption within defence.
AR field evaluation
The field evaluation investigated the potential delivery of enhanced Situational Awareness (SA) for both mounted (i.e. within a vehicle) and dismounted troops via the use of AR technology. Based upon Phase 1 feedback, four main sub-concepts were investigated:
- Augmented Navigation (dismounted): Participants following a set of waypoints, in both day and night conditions;
- Directional Alerts/Cueing (dismounted): Providing participants with directional information related to alerts/cueing;
- Proximity Alerts (dismounted): Providing stimulation to participants when encroaching/nearing an area or point of interest; and
- See-through Vehicle (mounted): Providing the participant with virtual camera views of the environment outside of the vehicle, based on the pointing direction of the visual display device.
The task performance of each participant was evaluated to provide an insight into how well the AR technology supported the participant, when compared against using a current baseline technology (e.g. using a map, compass and GPS locator for navigation).
Three separate AR technology types were assessed in the field evaluation, in order to investigate the relative merits of the different means of presenting information. First was audio, whereby headphones were used to convey information to the participant through the use of verbal or non-verbal audio signals. Second was haptic technology, which is a tactile feedback technology that simulates the sense of touch by applying forces, vibrations or motions to the participant. A haptic belt was used in the evaluation consisting of a number of haptic actuators that produce vibrations to convey information to the participant. Lastly, visual technology was used, whereby computer-generated inputs were augmented with real-world environment visuals provided by an AR-enabled sight or camera and tablet display to provide additional visual information to the participant.
Field evaluation results
The collected data provided both quantitative as well as qualitative indications on the performance of the technology. Each technology was evaluated using metrics based upon task performance, user workload, system usability and SA benefits and qualitative results were generated from observer and participant comments. The captured information provided the following insights:
- Navigation & Proximity Alerts: Route following using the audio visual AR concepts was more accurate in day and night conditions as compared to the baseline. Accuracy was improved using haptic during night time only. The results showed that the workload decreased for all the AR concepts and that audio and visual AR for day and night navigation and haptic AR for night navigation were rated in the top 10% of technologies tested for system usability.
- Cue-in: None of the AR technologies performed better than the baseline radio call for the cueing tasks. The visual and audio AR solutions were both liked by the users, but the trial showed how the lack of angular precision had a negative impact on task performance. The audio and visual AR did provide instant cues as compared to the radio call baseline, where it took approximately 10 seconds before a target was resolved.
- See-Through Vehicle: The results indicated that the system used caused an increased workload against the baseline, but scored higher on task performance and system usability.
The JFX3 project considered AR concepts across the full range of potential defence domains. The following opportunities were identified where AR technologies could be applied in the following ways to provide augmented targets and/or weapon effects to large scale training environments, using a combination of auditory, haptic and visual AR.
- Indirect Fire Augmentation: Simulated indirect fire detonations could be used to provide appropriate AR simulation associated with the detonation. The ‘own’ position of the soldier will need to be known to determine their relative position (heading and range) to the detonation. Different combinations of visual, auditory and haptic methods could be used to provide appropriate stimulations associated with the detonation effects.
- Entity Injection: Virtual targets, neutral or friendly forces could be provided via the use of AR. One of the greatest challenges will be the ‘registration’ of the virtual entity (i.e. does the entity appear to be in a plausible location, such as closely following the terrain for a land vehicle). Target occlusion will also be a challenge (i.e. is the entity hidden behind a physical feature or other entity?).
- Direct Fire ‘Crack Thump’ Augmentation: Auditory AR could be applied to simulate the sonic ‘crack’ of a round passing close by. Direct fire weapon effects simulation can detect near misses. However, because there is no actual round, there is no stimulation associated with the passage of the round. This concept could obtain information from the soldier’s tracking equipment which would detect the near miss. The critical aspect is the timing — the delay between the crack (from the round) and the thump (from the weapon fire) provides an indication of the range of the engagement.
AR for Maritime Training
A maritime AR training environment could be provided for those personnel who directly interact with the outside world (as opposed to those in the operations room who interact indirectly through sensors e.g. radar.) Personnel may be located at a ship’s bridge, upper deck mounted guns or a flight deck.
Synthetic entities (and weapon effects) could be injected that would be consistent with the stimulation provided to the operations room, to provide a collective training capability while at sea. Again, the challenge of correct entity registration will be important, as synthetic entities may need to appear to be floating on the real sea surface, and the effects of the own ship motion upon the AR viewpoint will need to be considered.
AR for Command and Staff Training
The use of AR is also being considered to support planning and operations within a headquarter environment (either in a building or deployed in field). An AR system such as a virtual ‘bird-table’ provides an interactive 3D representation of the battlefield to support briefing and shared situational awareness and the presentation of overlaid information. This will allow individuals to interact with the representation.
Barriers to AR adoption
A more in-depth study of the barriers was conducted as part of the JFX3 phase 2 project, including further discussions with key stakeholders. These discussions identified a variety of barriers, not all of which are unique to AR.
- Registration: a unique challenge for AR. It is the accuracy with which the synthetic overlay aligns with the outside world it is augmenting. Good registration is essential for defence work. It is rare in consumer AR technology, but can be achieved with thorough engineering.
- Atmospheric Isolation: ‘Atmospherics’ are the intangible cues in the environment which while hard to define, help the user get a feel for their surroundings. Any reduction in the intensity of these has an isolating effect on the user. Most AR technologies create a measure of atmospheric isolation since, by definition, they must insert some sort of device between the user and the environment in order to provide the augmentation.
- Immaturity of Eyewear: AR Head Mounted Display (HMD) and AR eyewear technologies are still quite immature. Since these are also the primary medium for delivering visual AR, this issue is the defining factor for the deployment and usefulness of visual AR.
- System Readiness versus Technology Readiness: Many types of AR are in fact already quite mature as individual technologies within their current markets. What is immature is their integration into full-blown real time systems or solutions that could be deployed for defence use.
- Pace of Obsolescence versus Pace of Acquisition: The pace of advancement in mobile technologies like AR vastly exceeds the normal defence acquisition cycle.
- Size, Weight Power and Robustness – Burden on the Soldier: This was one of the most clearly articulated barriers that must be addressed in order to gain acceptance for any new technology to be carried by the dismounted soldier. One stakeholder stated, “The soldier is not a Christmas tree. We can’t hang any more equipment on him.” If the soldier is required to carry additional kit this must be at the expense of an item that is being carried at the moment.
- Security: Areas of concern included how AR systems would handle the communication and storage of sensitive data; and policy and accreditation of AR equipment and software.
JFX3 has been an important project to understand more about the opportunities and barriers for AR in the defence industry. When used to support navigation, for example, AR facilitated more accurate route following, lower workload and enhanced system usability. One important provision, however, is that AR implementations are intuitive and easily used — without the need for any significant training — if they are to become a real asset for service personnel. The adoption of AR in the defence industry is still in its early stages but it is already becoming clear that the benefits could be maximised by using AR in conjunction with current baseline technologies and procedures. In fact, there is considerable scope to deploy a range of COTS capabilities with system integrators to create more innovative and effective AR solutions. We will no doubt see more of this in the years to come.