Research Report:

Augmented Reality
for Maintenance,
Repair and
Overhaul
(MRO)

This research has been sponsored by PTC.

RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

TABLE OF CONTENTS

Executive Summary 3

What is Augmented Reality? 4

Benefits of Using AR for MRO 5

Challenges of Using AR in Manufacturing 6

Augmented Reality Options – Hardware 7

Daqri Smart Glasses 8

Google Glass 9

Microsoft Hololens 10

Vuzix M100 & M300 Smart Glasses 11

Use Cases 12

Google Glass on the Audi A8 Production Line 12

Installation and Maintenance of Hauwei Sun2000-KTL Inverters 13

NLR & KLM Royal Dutch Training Concept 14

AR Elevator Maintenance at Thyssenkrupp 15

Augmented Reality for Maintenance, Repair and Overhaul 16

2
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

EXECUTIVE SUMMARY

The term Industry 4.0 denotes a cluster of technologies that’s poised to

fundamentally reshape manufacturing and bring about a new industrial
revolution. These include 3D printing, also known as additive manufacturing
(AM), the Industrial Internet of Things (IIoT), artificial intelligence (AI) and
mixed reality technologies, more commonly known as virtual reality (VR) and
augmented reality (AR). The two most common hurdles to implementing
these Industry 4.0 technologies are cost and integration.

However, the combination of increasing computing power and declining

hardware costs are rapidly diminishing the severity of the first hurdle. As
for meeting the second challenge—integration—that often comes down to
finding the right application where the technology truly shines. In the case of
AR, maintenance, repair and overhaul (MRO) applications are just that—which
is why we wrote this report.

In the following pages, readers will learn the benefits and challenges of using
AR for MRO. The positives include streamlining inspections, access to remote
expert assistance, and the ability to upskill on demand. These advantages
are contrasted with the challenges of using AR in an industrial environment,
which include employee reluctance to change, the difficulties of AR training
and potential interoperability issues.

Readers will also learn about the available options for augmented reality
from some of the world’s leading suppliers, including Google Glass, Microsoft
HoloLens and RealWear HMT-1. The report concludes with several case studies
of AR being used for MRO applications, all of which point to the idea that
augmented reality may offer a technological solution to the impending skills
gap in manufacturing.

3
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

WHAT IS AUGMENTED REALITY?

In addition to ‘augmented reality’ and ‘virtual reality’, you may have heard the
term ‘mixed reality’, which denotes a broader category. All three concepts can
be understood in terms of a continuum, with real environments at one end
and wholly virtual environments at the other. In between lies augmented
reality, as well as augmented virtuality. This in-between space in its entirety
constitutes mixed reality.

Mixed Reality (MR)

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Reality-Virtuality (AV) Continuum

The notion of a reality-virtuality continuum was first introduced more

than two decades ago by Paul Milgram, a professor of mechanical and
industrial engineering at the University of Toronto. To put the continuum
in a manufacturing context, think of the far left (Real Environment) as the
traditional factory floor: workers operating machines based on information
displayed on dials, gauges or screens.

The far right (Virtual Environment) is exemplified by the use of virtual reality in
factory floor planning.

It may be difficult to imagine examples of augmented virtuality in

manufacturing, but one possible application could be using handheld
peripherals to input commands, or testing ergonomics in VR factory-floor
planning.

4
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

BENEFITS OF USING AR FOR MRO

Even a passing familiarity with AR should give any manufacturing professional

ideas for implementing the technology on a production line. As a simple
example, Mitsubishi Electric has been developing maintenance-support
technology using augmented reality based on a 3D model that enables users
to confirm the order of inspection on an AR display and then enter inspection
results with their voice.

“We think that the most

useful application of
AR is maintenance in a
manufacturing environment,”
said a representative from the
Mitsubishi team developing
this technology. “Currently,
field workers confirm a
maintenance target using a
maintenance manual. The
process is time-consuming and tiring for the workers because the manual and
the maintenance target should be confirmed together.”

In addition to streamlining inspections by replacing a pen-and-paper

checklist with a voice-based system, augmented reality can enhance a worker’s
ability to perform maintenance routines by superimposing simple step-by-
step instructions on their field of view. This is the logical next step in being
able to see machine status at a glance using AR, which is a benefit that goes
beyond MRO applications.

Perhaps the most intriguing benefit of using AR for MRO is the ability for users
to invoke remote expert support when completing maintenance or repair
tasks. Field service often requires experts to travel to remote worksites, but the
telepresence afforded by AR means a single expert can service multiple sites
without ever having to leave the office.

Taken together, these

benefits constitute a kind
of upskilling on-demand,
enabling workers to perform
tasks that may go well beyond
their nonaugmented skillset.
That means reduced training
time and increased first-
time fix rates. While many of
these benefits apply to more
manufacturing operations
than just maintenance, AR is
particularly well-suited to meeting the demands of MRO applications.

5
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

CHALLENGES OF USING AR IN
MANUFACTURING

It’s easy to get swept up in the hype of new technology and consequently fail
to recognize the practical difficulties it presents. This is especially true in the
case of Industry 4.0 technologies—such as augmented reality—which tend
to be wrapped up with so many marketing buzzwords that one may wonder
whether there’s anything of substance to it at all. Now, while AR is certainly
more than a buzzword, there are nevertheless challenges for new adopters
which can be loosely divided into two categories: technical and organizational.

The technical challenges

for AR generally boil down
to the user interface, with
the need to improve field
of view (FOV), brightness,
display quality, latency,
etc. In the case of FOV, for
example, even the best AR
HUDs can only offer up to 90
degrees. Compare that to the
190-degree horizontal and
120-degree vertical FOVs for
normal human vision and the gap between where the technology is now and
where it needs to go becomes obvious.

The principal organizational

challenges for AR are a
lack of knowledge about
the technology and
an insufficient level of
digitalization within the
company. The first challenge
is somewhat ironic, given
the new possibilities for
training and education that
augmented reality brings to
the table. However, unlocking
that potential requires one
to recognize it in the first place, which is difficult in the absence of first-hand
experience. The second challenge—insufficient digitalization—is an issue for
many Industry 4.0 technologies, including augmented reality. However, it’s
particularly problematic for AR because of how much the technology relies
on data collection and analysis. Without a sufficient number of assets sharing
information, adding AR capabilities will do little to improve plant efficiency.

6
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

AUGMENTED REALITY OPTIONS –

HARDWARE

According to a recent analysis by Research and Markets, the global AR/VR

market will reach $94.4 billion USD by 2023. Although VR devices currently
make up a larger portion of that market, the analysis predicts that the market
for AR devices will grow faster, with a CAGR of 73.8 percent. All this growth
entails some uncertainty regarding which AR hardware options will still be
supported a decade from now. That’s why this report focuses on surveying the
current AR offerings from some of the best-known suppliers. Check out the AR/
VR section on engineering.com for the most up-to-date information.

7
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

DAQRI SMART GLASSES

The latest iteration of the company’s augmented reality hardware, DAQRI’s
Smart Glasses are designed for professional use. The unit is based on a 6th
generation Intel Core m7 processor and uses dual LCoS optical displays, which
give it a 44° diagonal FOV. It comes equipped with Worksense Standard,
DAQRI’s suite of apps for AR tasks. Detailed specs for DAQRI Smart Glasses
appear below.

Processor 6th Gen Intel Core m7 (Up to 3.10 GHz)

Optics Dual LCoS Optical Displays
Resolution: 1360 x 768
Frame Rate: 90 fps
Connectivity WiFi 802.11 A/B/G/N/AC 2.4/5 GHz
Bluetooth
Battery Li-ion 5800 mAh
Storage 64 GB SSD
I/O Ports 2 USB 3.1 Type C
3.5mm Headphone Jack
Audio 2 Microphones with Active Noise Cancellation
Depth Sensor Camera Range: 0.4mm to 4m
Resolution: 640 x 480
Frame Rates: 30, 60, 90 fps
Color Camera RGB 1080p HD, 30 fps
AR Tracking Camera 166° Diagonal Wide-Angle Fisheye Lens
Resolution: 640 x 480
Frame Rate: 30 fps

8
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

GOOGLE GLASS
While it failed to find purchase in the consumer market, Google’s foray into
augmented reality has seen considerably more success in medical and
manufacturing applications. The device features some ambitious technology,
including bone conduction for audio output and a prism projector display. The
successor to the Explorer Edition, Google Glass Enterprise Edition eschews the
bone conduction tech, but also makes several improvements over previous
iterations, including an Intel Atom processor and a barometer. Detailed
specifications for Google Glass Enterprise Edition appear below.

Processor Intel Atom (32) -model unspecified

Optics 640 x 360 Himax HX7309 LCoS display
Connectivity Dual-band 2.4+5 GHz 802.11 a/b/g/n/ac
Bluetooth
Battery 780 mAh
Storage 32GB
I/O Ports Micro USB
Audio Speaker -details unspecified
Sensors Barometer
Capacitive head sensor
Hinge sensor
Assisted GPS & GLONASS
Cameras Supports 5MP stills and 720p video -details
unspecified

9
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

MICROSOFT HOLOLENS
Billed as the world’s first full untethered holographic computer, the
HoloLens is available in both a Development Edition and a Commercial Suite
configuration. Companies based in North America also have the option of
renting a HoloLens through a Microsoft partner. The current version of the
HoloLens has a 35° FOV, but a recent patent from Microsoft for a MEMS laser
scanner could increase the FOV for the next generation of HoloLens to 70°.
Detailed specifications for the HoloLens appear below.

Processor Intel 32 bit architecture with TPM 2.0 support

Custom-built Microsoft Holographic Processing
Unit (HPU 1.0)
Optics See-through holographic lenses (waveguides)
2 HD 16:9 light engines
Holographic Resolution: 2.3M total light points
Holographic Density: >2.5k radiants (light points
per radian)
Connectivity Wi-Fi 802.11ac
Bluetooth
Battery 2-3 hours of active use
Up to 2 weeks of standby time
Fully functional when charging
Storage 64GB Flash
I/O Ports 3.5mm Headphone Jack
Micro USB 2.0
Audio Built-in speakers
4 microphones
Cameras/Sensors 1 IMU
4 environment understanding cameras
1 depth camera
1 2MP photo/HD video camera
1 ambient light sensor

10
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

VUZIX M100 & M300 SMART GLASSES

Founded in 1997, Vuzix has a long history in the augmented reality industry.
Its current offerings suitable for industrial applications are the M100 and M300
Smart Glasses. The M300 is marketed as having enhanced functionality, and
there are some important differences between the two. For example, the
diagonal FOV on the M100 is 15° and 20° on the M300. Detailed specs for both
models appear below.

M100 M300
Processor OMAP4430 @ 1GHz Dual Core Intel Atom CPU
Optics WQVGA color displays 16:9 nHD color display 16:9
24-bit color 24-bit color
Connectivity Wi-Fi 802.11 b/g/n Wi-Fi b/g/n/ac
Bluetooth Dual-Band 2.4/5 GHz
Bluetooth
Battery 600mha internal battery 160mAh internal battery
3800mha external battery 860mAh external battery
pack
Storage 4GB internal flash 64GB internal flash
memory memory
Micro SD support up to
32GB
I/O Ports MicroUSB MicroUSB 2.0 HS
Audio Ear speaker and noise Ear speaker and noise
cancelling microphone cancelling microphone
Cameras/Sensors 5 MP Stills 10 MP Stills
1080p Video, 16:9 1080p Video
3 DOF gesture engine Auto-Focus
(L/R, N/F) Optical Image
Proximity & Ambient Stabilization
Light Sensors Flash/Scene Illumination
Inward & Outward Facing
Proximity Sensors

11
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

USE CASES

Although AR is still a relatively new technology in the manufacturing space,

there are already numerous examples of its benefits in a broad range of
applications. Many of these are within the space of maintenance, repair and
overhaul. This report focuses on four use cases in particular.

GOOGLE GLASS ON THE AUDI A8 PRODUCTION LINE

In 2015, researchers from Heilbronn University in Germany ran a four-month

project on the AUDI A8 production line using the Explorer Edition of Google
Glass. The target task was the calibration of the drive assistance system testing
bay, which fell to workers on the Monday morning shift. All the sensors and
actuators on this system have to be checked at various intervals (e.g., weekly,
monthly, yearly), further complicating the calibration.

Prior to the study,

workers calibrated
the testing bay
using paper-based
documentation.
Interestingly,
although the intent
was to replace this
pen-and-paper
system with one
based on AR, the
researchers elected to
leave the calibration
workflow unchanged
after determining
that it was already “adequately optimized.” Normally, the calibration
involves consulting data displayed on various screens in the testing bay and
documenting performance by stamping a protocol book. The researchers were
able to replace this approach with a voice-based system, encouraging users to
“think out loud” while using Google Glass.

As a result, according to the researchers, “Any employee independent from

his training level now is able to perform the calibration of the driver assistance
system testing bay in an acceptable time span.”

12
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

INSTALLATION AND MAINTENANCE OF HAUWEI

SUN2000-KTL INVERTERS

In 2016, the electronics giant Hauwei partnered with Inglobe Technologies

to develop an AR solution for the installation and maintenance of the former
company’s SUN2000-KTL inverters. According to Hauwei, these units are
spread all over the world, potentially requiring technicians to travel significant
distances in order to maintain them. Moreover, the variety of issues that a
technician might encounter means that they are generally required to be
familiar with more procedures than they can possibly remember. Taken
together, these factors point to augmented reality as an obvious solution.

For Inglobe
Technologies, the
primary challenges
for this project
were to create an
application that
could rely entirely
on natural feature
tracking and which
could reliably
function outdoors,
despite varying light
levels. On top of that,
Inglobe Technologies
never had direct
access to a SUN2000-KTL inverter. Nevertheless, the two companies were
able to create a simple interface that would work for Chinese and American
technicians alike.

Once the user selects a language (English or Mandarin), the application can
automatically recognize a nearby inverter and display a list of procedures on
screen. These can be assigned by a maintenance manager or initiated by
engineers in the field. Procedures can then be followed in a linear fashion or
accessed as required.

According to Rhonda Truitt, Director of US R&D Technical Communication

Innovation & Best Practices at Hauwei, “Our customers have responded
overwhelmingly positive about this experience. I am confident that we will
see a sharp decline in installation errors due to this AR experience for post-
installation checks.”

13
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

NLR & KLM ROYAL DUTCH TRAINING CONCEPT

In 2017, the Netherlands Aerospace Centre (NLR) and KLM Royal Dutch Airlines
collaborated on a training concept for aircraft maintenance engineers. Groups
of students were taught how aircraft air conditioning systems work in a typical
classroom setting. However, during the training, students and instructors both
wore HoloLens systems in order to view detailed 3D models of A/C systems.
This allowed students to walk around and through the system, and also
collaborate on tasks and ask “hands-on” questions of the instructor.

According to NLR,
this approach to
training yielded
significantly better
results—particularly
in terms of student
engagement—
compared to
conventional
techniques. “[I]n
the virtual world
it is easier for an
instructor to look
over the shoulders of
the trainees than in
the real world, enabling him to give feedback,” said Harrie Bohnen, Manager
of NLR’s Training, Simulation and Operator Performance Department. “The
training can be given anywhere, and practical exercises can be performed in
a virtual aircraft instead of a real one. You no longer need the actual aircraft,
which increases its availability and reduces wear and tear.”

14
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

AR ELEVATOR MAINTENANCE AT THYSSENKRUPP

The German elevator manufacturer, thyssenkrupp, was one of the early

adopters of AR for MRO when it began using the HoloLens in 2016. The
company announced plans to equip over 24,000 service technicians with AR
capabilities, enabling them to visualize and identify problems before they
arrive at the job site, in addition to giving them remote, hands-free access to
technical and expert information on-site.

According to the
company, initial
field trails saw
service maintenance
interventions being
performed up to four
times faster. This is
no doubt in part due
to the integration of
HoloLens with MAX,
thyssenkrupp’s IoT-
enabled predictive
maintenance
solution, launch back
in 2015.

“By utilizing the out-

of-the-box Skype
experience without
any additional
development
required,
thyssenkrupp’s
24,000 service
engineers can now
do their jobs safer
and more efficiently,”
noted Scott Erickson,
General Manager for
Microsoft HoloLens.
“Triaging service
requests ahead of the
visit and getting hands-free remote holographic guidance when on site has
reduced the average length of thyssenkrupp’s service calls by up to 4x.”

15
RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

AUGMENTED REALITY FOR

MAINTENANCE, REPAIR AND OVERHAUL

It’s easy to dismiss the technologies that fall under the broad heading of the
fourth industrial revolution as being overhyped or years’ away from being
practical, but as augmented reality in the context of maintenance, repair
and overhaul applications demonstrates, there are pragmatic reasons for
taking AR and other Industry 4.0 technologies seriously. The decreasing cost
of AR hardware combined with the growth of the overall market means that
augmented reality solutions are only going to become more accessible over
time. As more and more manufacturers recognize the value of AR for MRO,
having AR capabilities will become less a competitive advantage and more a
necessity.

How long can you afford to wait?

This research has been sponsored by PTC.

Industrial AR connects users across digital and physical worlds.
It will do more than just change products, operations, and
services – it’ll transform business.
To learn more, visit:
https://www.ptc.com/vuforia
16