1.

INTRODUCTION

1
1.1. Introduction

Mixed Reality also referred as ‗Hybrid Reality‘ is a combination of both virtual

reality and physical reality which creates a completely new environment where
both physical and digital objects co-exist & interact in real time. Due to the
advancement in technologies such as computer vision, graphical processing
power, input systems and display technologies it is now possible to encompass
both the physical world and the virtual world via immersive technology. Mixed
reality has its roots in the early 1990‘s when the first immersive mixed reality
system called Virtual Fixtures Platform was developed at the US Air Force‘s
Armstrong Laboratories. Though the term Mixed Reality was first coined 4 years
later in a 1994 paper by Paul Milgram & Fumio Kishino, ―A Taxonomy of Mixed
Reality Visual Displays‖. In this paper, they elaborated the concept of Virtuality
Continuum & how displays are categorized in the taxonomy. Mixed Reality
basically breaks down the barriers between physical and virtual reality. It
comprises of all the spectrums of situations which spans the continuum
between virtual reality and actual reality. It includes augmented reality where
the virtual augments the real, & augmented virtuality, where the real augments
the virtual and the various configurations which fall within virtual reality & Real
Environments.

1.2. History

Mixed reality is the result of blending the physical world with the digital world.
Mixed reality is the next evolution in human, computer, and environment
interaction and unlocks possibilities that before now were restricted to our
imaginations. It is made possible by advancements in computer vision, graphical
processing power, display technology, and input systems. The term mixed
reality was originally introduced in a 1994 paper by Paul Milgram and Fumio
Kishino, ―A Taxonomy of Mixed Reality Visual Displays." Their paper introduced
the concept of the virtuality continuum, and focused on how the categorization
of taxonomy applied to displays. Since then, the application of mixed reality goes
beyond displays. It also includes environmental input, spatial sound, and
location.

2
 2. VIRTUAL REALITY Vs
AUGMENTED REALITY Vs MIXED
REALITY

3
2.1. Augmented Reality

Augmented reality (AR) adds digital elements to a live view often by using the
camera on a smartphone. Examples of augmented reality experiences include
Snapchat lenses and the game Pokémon Go. In augmented reality, users see and
interact with the real world while digital content is added to it. If this sounds
unclear think of Pokémon Go – millions of people all over the world have been
rushing with their smartphones in search for small virtual creatures. That‘s the
most vivid example of augmented reality.

If you own a modern smartphone, you can easily download an AR app and try
this technology. There‘s a different way to experience augmented reality, though
with special AR headsets, such as Google Glass, where digital content is
displayed on a tiny screen in front of a user‘s eye.

2.2. Virtual Reality

Virtual reality (VR) implies a complete immersion experience that shuts out the
physical world. Using VR devices such as HTC Vive, Oculus Rift or Google
Cardboard, users can be transported into a number of real-world and imagined
environments such as the middle of a squawking penguin colony or even the
back of a dragon.

This technology immerses users in a completely virtual environment that is

generated by a computer. The most advanced VR experiences even provide
freedom of movement – users can move in a digital environment and hear
sounds. Moreover, special hand controllers can be used to enhance VR
experiences.

You need to wear a special VR headset to experience virtual reality. Most VR

headsets are connected to a computer (Oculus Rift) or a gaming console
(PlayStation VR) but there are standalone devices (Google Cardboard is among
the most popular) as well. Most standalone VR headsets work in combination
with smartphones – you insert a smartphone, wear a headset, and immerse in
the virtual reality.

4
2.3. Mixed Reality

In a mixed reality (MR) experience, which combines elements of both AR and VR,
real-world and digital objects interact. Mixed reality technology is just now
starting to take off with Microsoft‘s HoloLens one of the most notable early mixed
reality apparatuses.

This is the most recent development in reality technologies that sometimes

causes confusion, primarily because different experiences are called so. Mixed
reality that starts with the real world – virtual objects are not just overlaid on the
real world but can interact with it. In this case, a user remains in the real-world
environment while digital content is added to it; moreover, a user can interact
with virtual objects. This form of mixed reality can be co nsidered an advanced
form of AR. Mixed reality that starts with the virtual world – the digital
environment is anchored to and replaces the real world. In this case, a user is
fully immersed in the virtual environment while the real world is blocked out.
Sounds like virtual reality, right? In fact it does, but the digital objects overlap
the real ones whereas in conventional VR the virtual environment isn‘t
connected to the real world around a user.

If we look at the way smartphones changed our lives, they took something that
was already there, the internet, and basically, put it into our pockets. They made
the world much more connected and brought an unprecedented level of
convenience in our lives. Mixed reality will have an even more significant impact
because it will drastically alter the way we interact with the world. Since we will
be free from the shackles of the physical restrictions, it will revolutionize our
existence in the world. As an example, let‘s take a look at the business world. A
lot of companies, especially large corporations, have offices in various locations
around the country and even on different continents. Sometimes it is required to
travel to a particular place to participate in a meeting. This will become a thing
of the past. Thanks to mixed reality, you could be on another side of the planet,
yet it will look and feel as if you are all in the same room together.

5
In the education world, it will make it easier to learn subjects by making them
interactive, and you will have an AI-powered assistant that will always be
available who is aware of what you are trying to learn in addition to the digital
overlays that can deliver knowledge in real-time. Keep in mind, this is not merely
limited to the classroom. Mixed reality software development is working on
democratizing expertise and can help you in everyday life situations. For
example, if your car breaks down and you have no idea how to fix it, you can use
MR to find out where all of the car parts are located, and it can also provide you
with instruction on how to fix it. We can provide a lot more mixed reality
examples, but the biggest takeaway is that MR will take what we have achieved
with AR to a whole new level and completely revolutionize how we interact with
the world.

6
3. WORKING OF MIXED REALITY

7
3.1. How does mixed reality works?

With the development of the technology is shrouded in secrecy by the major

players, all that is known on the operations of the technology is that it uses
virtual reality and augmented reality as well as space and coordinates. Magic
Leap is the company at the forefront in the development of mixed reality and
they have remained tight lipped about the finer details of what they are doing or
developing. The scanty details we know however, suggests that just like
Hololens, mixed reality will use a projector for displaying images on
semitransparent materials which will then be reflected to the eye with the help of
beam-splitting technology.

8
4. MIXED REALITY DISPLAY
TECHNOLOGIES

9
Cave automatic virtual environment

CAVE (Cave Automatic Virtual Environment) is a virtual reality environment

consisting of a cube-shaped VR room in which the walls, floors and ceilings are
projection screens. The user typically wears a VR headset or head-up display
(HUD) and interacts through input devices such as wands, joysticks or data
gloves. The CAVE is contained within a larger room that must be absolutely dark
when the CAVE is in use. Three-dimensional images within the cave appear to
float in mid-air. The viewer, whose headgear is synchronized with the projectors,
can walk around an image to study it from all angles. Sensors within the room
track the viewer's position to align the perspective correctly.

Head-up display

A head-up display or heads-up display, also known as a HUD,is any transparent

display that presents data without requiring users to look away from their usual
viewpoints. The origin of the name stems from a pilot being able to view
information with the head positioned "up" and looking forward, instead of angled
down looking at lower instruments. A HUD also has the advantage that the
pilot's eyes do not need to refocus to view the outside after looking at the
optically nearer instruments.

Head-mounted display

A head-mounted display (HMD) is a display device, worn on the head or as part

of a helmet that has a small display optic in front of one (monocular HMD) or
each eye (binocular HMD). An HMD has many uses including gaming, aviation,
engineering, and medicine. Head-mounted Displays are the primary components
of virtual reality headsets. There is also an optical head mounted
display (OHMD), which is a wearable display that can reflect projected images
and allows a user to see through it.

10
5. WINDOWS MIXED REALITY

11
Windows Mixed Reality is a mixed reality platform introduced as part of
the Windows 10 operating system, which provides holographic and mixed reality
experiences with compatible head-mounted displays.

Its flagship device, Microsoft HoloLens, was announced at the "Windows 10: The
Next Chapter" press event on January 21, 2015 .It provides a mixed reality
experience where a live presentation of physical real-world elements is
incorporated with that of virtual elements (referred to as "holograms" by
Microsoft such that they are perceived to exist together in a shared
environment. A variant of Windows for augmented reality computers (which
augment a real-world physical environment with virtual elements) Windows
Mixed Reality features an augmented-reality operating environment in which
any Universal Windows Platform app can run.

The platform is also used for virtual reality headsets designed for use on
the Windows 10 Fall Creators Update, which are built to specifications
implemented as part of Windows Mixed Reality, but lack support for holographic
experiences.

5.1. Microsoft HoloLens

The HoloLens is a head-mounted display unit connected to an adjustable,

cushioned inner headband, which can tilt HoloLens up and down, as well as
forward and backward. To wear the unit, the user fits the HoloLens on their
head, using an adjustment wheel at the back of the headband to secure it
around the crown, supporting and distributing the weight of the unit equally for
comfort,] before tilting the visor towards the front of the eyes.

The front of the unit houses many of the sensors and related hardware,
including the processors, cameras and projection lenses. The visor is
tinted; enclosed in the visor piece is a pair of transparent combiner lenses, in
which the projected images are displayed in the lower half. The HoloLens must
be calibrated to the interpupillary distance (IPD), or accustomed vision of the
user.

Along the bottom edges of the side, located near the user's ears, are a pair of
small, red 3D audio speakers. The speakers, competing against typical sound
systems, do not obstruct external sounds, allowing the user to hear virtual
sounds, along with the environment. Using head-related transfer functions, the

12
HoloLens generates binaural audio, which can simulate spatial effects; meaning
the user, virtually, can perceive and locate a sound, as though it is coming from
a virtual pinpoint or location.

On the top edge are two pairs of buttons: display brightness buttons above the
left ear, and volume buttons above the right ear. Adjacent buttons are shaped
differently—one concave, one convex—so that the user can distinguish them by
touch.

At the end of the left arm is a power button and row of five, small individual LED
nodes, used to indicate system status, as well as for power management,
indicating battery level and setting power/standby mode. A USB 2.0 micro-
B receptacle is located along the bottom edge. ] A 3.5 mm audio jack is located
along the bottom edge of the right arm.

13
5.2. Immersive headsets
In October 2016 during a hardware event, Microsoft announced that multiple
OEMs would release virtual reality headsets for the Windows Holographic
platform, based on Microsoft reference designs enabling room-scale virtual
reality without external sensors or components. In January 2017, prototypes
were presented at Consumer Electronics Show for release later in the year, and
Microsoft later announced that it planned to release development kits for such
headsets during the Game Developers Conference. These devices would be
supported by the Windows 10 "Creators Update". At the Game Developers
Conference in 2017, Microsoft stated that it intended to support Windows Mixed
Reality headsets on Xbox One in 2018, specifically noting the capabilities of the
then-upcoming Xbox One X hardware revision, but the company later stated
that it was initially focusing on PC platforms first, and that it wanted to focus on
wireless VR solutions for consoles.

In October 2017, Microsoft officially launched Windows Mixed Reality and a

lineup of third-party headsets for use with the Windows 10 "Fall Creators
Update" (including a launch lineup of headsets from Acer, Dell, HP, and Lenovo,
and future products from Asus and Samsung), officially referred to as Windows
Mixed Reality immersive headsets. Unlike HoloLens, these devices are only
compatible with virtual reality software, but the underlying ecosystem is referred
to as Windows Mixed Reality regardless of experience. All Immersive headsets
feature integrated motion tracking (eschewing standalone sensors), and contain
cameras that can be used to track handheld motion controller accessories,
which may be bundled with the headset, or distributed as optional accessories.

Immersive headsets are currently compatible with mixed reality software

obtained from Microsoft Store, universal apps, and Steam VR-compatible
software. The ability to run desktop software was added on the Windows 10
"May 2019 update.

14
5.3. Motion controllers

Motion controllers are hardware accessories that allow users to take action in
mixed reality. An advantage of motion controllers over gestures is that the
controllers have a precise position in space, allowing for fine grained interaction
with digital objects. For Windows Mixed Reality immersive headsets, motion
controllers are the primary way that users will take action in their world.

15
Windows Mixed Reality motion controllers offer precise and responsive tracking
of movement in your field of view using the sensors in the immersive headset,
meaning there is no need to install hardware on the walls in your space. These
motion controllers will offer the same ease of setup and portability as Windows
Mixed Reality immersive headsets.

Features

 Optical tracking
 Trigger
 Grab button
 Thumbstick
 Touchpad

16
6. APPLICATIONS

17
Education
Mixed reality technologies are being used within the education industry to both
enhance students‘ ability to learn and take in information. It also gives the
students the opportunity to personalize the way they learn.

Using 3D projections and simulations, students can interact with and

manipulate virtual objects in order to study them in a way that is relevant to
themselves and their studies. By inserting three-dimensional objects into a
classroom as a means of gauging the size, shape, or other features of a defined
―virtual‖ object, students can gain a deeper sense of understanding as to what it
is they‘re studying.

Some ways that MR can help in the classroom?

 Interact with the environment in an immersive experience.
 Touch and manipulate objects.
 It is an engaging and fun way of learning.
 MR can teach any kind of subject.

Engineering

Mixed reality in engineering is slowly but surely becoming a game-changer. From

3D modeling and virtual sculpting to remote repair guidance and project
monitoring apps. There are various ways in which the engineering sector has
begun to take advantage of mixed reality devices.

Some benefits in Engineering?

 Real-time simulation of engineering processes.

 Use MR with an industrial IoT device to monitor services.
 Engineering training.

18
Entertainment

VR entertainment has been around for decades and because of this, it‘s the most
acclimatized and most widely available of the three technologies. Augmented
reality games and apps are coming out thick and fast since the success of
Pokémon Go and there‘s little reason to suspect they‘ll disappear out anytime
soon.

But the mixed reality in entertainment is not only reserved for gaming.

AR and VR technology is improving so much that the film and big screen
industry are after the vision of incorporating them into their movies. The idea of
interactive storytelling which is a mix between a game and movie will open up
lots of new doors for mixed reality. An example is the new Netflix series, you vs.
Wild with adventure survivalist Bear Grylls, which lets you make decisions and
change endings.

Mixed reality entertainment is already here with companies like Magic Leap,
Lucas film, and Industrial Light And Magic all looking to delve into mixed reality
entertainment. For example, Magic Leap‘s is embracing mixed reality in cinema
by using what they call a Dynamic Digitised Light field Signal. Magic Leap‘s tech
projects images directly into the eye, without the need for it to bounce off an
object and then head towards the eyes. This tricks the brain into thinking the
object is there when in reality, it is a projection.

Healthcare

When it comes to healthcare, mixed reality technologies have many potential

applications. The most obvious is training and education. An example is the
over-the-shoulder surgeries, where surgical students can be taught remotely by
experts as they perform surgeries in real-time.

Another example is interactive learning. Topics like anatomy with mixed reality
technology can be used to map the different layers of the human body. Being
able to produce three-dimensional models of the anatomy complete with
information accessible by just a simple gesture could change the way health care
and medicine is taught.

19
MR will also transform the way in which medical students learn, using three-
dimensional holograms in a virtual environment rather than two-dimensional
diagrams from medical textbooks in base reality.

Military training

Combat reality is simulated and represented in complex, layered data through

HMD.Military training solutions are often built on commercial off-the-
shelf (COTS) technologies, such as Virtual Battlespace 3 and VirTra, both of
which are used by the United States Army. As of 2018, VirTra is being used by
both civilian and military law enforcement to train personnel in a variety of
scenarios, including active shooter, domestic violence, and military traffic
stops. Mixed reality technologies have been used by the United States Army
Research Laboratory to study how this stress affects decision-making. With
mixed reality, researchers may safely study military personnel in scenarios
where soldiers would not likely survive.

In 2017, the U.S. Army was developing the Synthetic Training Environment
(STE), a collection of technologies for training purposes that was expected to
include mixed reality. As of 2018, STE was still in development without a
projected completion date. Some recorded goals of STE included enhancing
realism and increasing simulation training capabilities and STE availability to
other systems.

It was claimed that mixed-reality environments like STE could reduce training
costs, such as reducing the amount of ammunition expended during training. In
2018, it was reported that STE would include representation of any part of the
world's terrain for training purposes. STE would offer a variety of training
opportunities for squad brigade and combat teams, including Stryker, armory,
and infantry teams. STE is expected to eventually replace the U.S. Army's Live,
Virtual, and Constructive – Integrated Architecture (LVC-IA).

Remote working

Mixed reality allows a global workforce of remote teams to work together and
tackle an organization's business challenges. No matter where they are
physically located, an employee can wear a headset and noise-canceling
headphones and enter a collaborative, immersive virtual environment. As these
applications can accurately translate in real time, language barriers become

20
irrelevant. This process also increases flexibility. While many employers still use
inflexible models of fixed working time and location, there is evidence that
employees are more productive if they have greater autonomy over where, when,
and how they work. Some employees prefer loud work environments, while
others need silence. Some work best in the morning; others work best at night.
Employees also benefit from autonomy in how they work because of different
ways of processing information. The classic model for learning styles
differentiates between Visual, Auditory, and Kinesthetic learners.

Machine maintenance can also be executed with the help of mixed reality. Larger
companies with multiple manufacturing locations and a lot of machinery can
use mixed reality to educate and instruct their employees. The machines need
regular checkups and have to be adjusted every now and then. These
adjustments are mostly done by humans, so employees need to be informed
about needed adjustments. By using mixed reality, employees from multiple
locations can wear headsets and receive live instructions about the changes.
Instructors can operate the representation that every employee sees, and can
glide through the production area, zooming in to technical details and explaining
every change needed. Employees completing a five-minute training session with
such a mixed-reality program has been shown to yield the same results as
reading a 50-page training manual.

21
7. CONCLUSION

22
Mixed reality comes along with immeasurable potential of changing how people
work, interact and communicate with the real world. As the technology matures
and the accompanying devices get smaller, it is expected that various sectors
and industries will adopt the use of mixed reality in their day to day operations.

But while the technology develops and gets better each day, topics such as the
acceptable social use and etiquette will also have to be addressed. For instance,
how will an employee with mixed reality device interact with fellow employees at
the workplace Or how are the social norms expected to change so as to
accommodate devices such as holograms which can only be seen by a few
people.

Also, since mixed reality experiences take place in the physical world,
individuals are likely to react to them more emotionally than to a mere digital
experience. As the technology develops, it is imperative for developers as well as
the intended users to be aware of such issues and be ready to address them
accordingly.

23
8. References

1. R. Freeman, A. Steed and B. Zhou, Rapid Scene Modeling, Registration

and Specification for Mixed Reality Systems Archived 2007-02-06 at
the Wayback Machine Proceedings of ACM Virtual Reality Software
and Technology, pp. 147-150, Monterey, California, November 2005.

2. Jump up to P. Milgram and A. F. Kishino (1994). "Taxonomy of Mixed

Reality Visual Displays". IEICE Transactions on Information and
Systems. pp. 1321–1329. Retrieved 2013-10-17.

24