Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Augmented Reality
Augmented reality, or AR, is when real reality, or RR, is enhanced with virtual
images that appear to coexist with the real world.
Virtual Reality
Virtual reality, or VR, is when you’re immersed in a purely virtual world
Mixed Reality
5
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Technology Definitions
6
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Ontology of AR / VR
Father of Augmented Reality,
Steve Mann
“Ultimate Display”, a VR
helmet with two CRT
screens, was created by
Link trainer: Invented by Sensorama: First VR machine Ivan Sutherland. The
Edwin Link for training developed by Morton Heilig, two CRT screens
military pilots, using gives visual and sensory enabled the viewer to
simulation perceptions of virtual reality, experience the 3D
including vibrations, smell images
and touch.
9
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
2nd WAVE: Industrial & Military Applications
1966-68 1972
11
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
4rd WAVE: Next computing platform
The 4th wave of virtual and augmented reality began, when
Facebook bought, the VR pioneering company Oculus for $ 2
Billion US Dollars.
12
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Predictions and hype present and in the past
The only outcome I can make is …predictions are always wrong…but they can
indicate media hype and technological advances
Source: Frost & Sullivan, 4th Wave in:Bauer (1996), OVUM in Leston et al. (1996),
Machover (1996)
VRML, 1995
The Sword of Damocles :
VRML was billed as a 3-
Ivan Sutherland , 1968
Power Glove : Nintendo, D alternative to HTML.
1989, price - $75 The idea was that users
could interact freely with
3-D worlds on the
internet, described by
text and interpreted by
modeling software.
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Why AR & VR didn't get broad adoption?
• Technical imperfection
• Cost
• Bulky & Bad design
• Failed to deliver value to the end user
19
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
AR & VR Ecosystem : Cont.
20
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
AR & VR Ecosystem : Cont.
DISPLAY HARDWARE: [HMD]: AlterGaze, AntVR,
FOVE, AVEGANT, HTC, Oculus, Sony, Panasonic,
VRVANA, Razer, SENSICS, Immersion VRELLA, Light
& Shadows & Mindmaze.
21
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
VR Ecosystem
Number of VR companies Investments in VR
VR Investment landscape
23
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape
There are more than 60 companies in Russian AR/VR market.
Mostly these companies are digital studios. However, there are high-tech companies with proprietary technology
developed in-house.
24
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape: Cont.
25
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape: Cont.
26
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Russian Landscape: Cont.
AR / VR continuum: controllers / 3D visualization & 3D content creation / haptic & sensory feedback / images
& video search and analysis / gesture recognition, computer vision etc.
27
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Current Market Analytics :- Revenue Users and Investments
28
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Current Market Analytics :- Market Share
29
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Market Analytics : Revenue generation and adoption
Based on the information above, we can conclude that VR HMD will drive the industry forward.
However, content will be king. That’s why a lot of VR companies invest heavily in VR studios, create SDK
for developers. And recently Oculus Rift announced that hundreds of games are in the process of the
development and thousands of developers are involved in this process.
31
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Gaming market dissection
33
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
VR/ AR Market drivers and projections
• Physical Risk. Any discrepancy between what the user sees and feels
could lead to motion sickness.
• Security Risks. For example, if users are able to modify the appearance
of their avatars, they could potentially impersonate other users.
• Price. Many experts think that the limiting factor for virtual reality is
going to be the price. The reality is a bit more complicated.
1. Display Resolution: 1080p 1. The GPU needs 50 bytes per polygon, which then is
2. Frames per second: 60-90 multiplied for 12 million polygons, would be 600
3. Depth of resolution: 24 bit Megabytes per second, with 60 frames per second.
4. Texture information: 12 2. In order to calculate the Bandwidth, it will be : 600
million per scene megabytes x 60 frames/sec = 36 GB/sec [Pass 1]. For 90
5. Latency: <20 ms frames/sec it would be 54 GB/sec
6. Pixel Density: 300 dpi 3. According to OpenGL and DirectX best practices, a
minimum of 3 passes on each polygon must be made.
This is done in order to draw the polygon, emulate its
animation properties and to display its texture
NVIDIA
Tesla GPU Pascal GPU
37
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Use case: GPU horsepower needed - Cont.
Information given: Calculation:
1. Display Resolution: 1080p
2. Frames per second: 60-90
3. Depth of resolution: 24 bit 4. Therefore, the bandwidth required by the GPU would be 36 GB/sec x
4. Texture information: 12 million per 3 = 108 GB/sec. For 90 frames it would be 54 GB/sec x 3 = 162 GB/sec
scene
5. Latency: <20 ms 5. Additional considerations: Texture characteristics and attributes, such
6. Pixel Density: 300 dpi as shining, shading, colour and material need to be analysed and
CPU vs GPU Performance incorporated within these parameters
6. In order to rasterize, 1 polygon, it would need around 100 floating-
point operations. Therefore, for 12 million polygons, it would be:
12,000,000 x 100 flops x 60 frames/sec x 3 raster passes,
it would equate to around: 216 GFLOP/ sec
For 90 fps – 324 GFLOP/s
Therefore, energy stored in the battery for scenario 1 would be: 145
Wt*h
Scenario 2: 290 Wt*h
Scenario 3: 1160 Wt*h
For commercial implementation, Lithium Ion gives best performance
and is supported by wide industry standards.
The typical energy density stored in a mobile phone battery is
between 0,9 – 2,3 MJ/L. This equates to 740 Wt*h/L.
39
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Use case: Battery required to power GPU for realistic scene
Scenario 2:
Volume = 0,000391 m3
Equal dimensions = 0,073m
Scenario 3:
Volume = 0,001567 m3
Equal dimensions = 0,12 m
Outcome:
1. For mobile VR system with realistic graphics
working for 1 hour we will need battery with 6
cm x 6 cm x 6 cm dimensions.
2. For mobile VR HMD working at least 8 hours
we will need a battery with 12 cm x 12 cm x 12
cm.
Energy density of Lithium Ion batteries over the years, Battery University, 2014
40
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Complexities within VR & AR
Based on the data obtained from AMD, with Oculus being the benchmark
IMAGE DISPLAY standard, it is evident from the information table above, that a ordinary
Graphics IMAGE RENDERING gaming PC would be unable to suffice the demanding computations for
Sub- DRIVER PROCESSING on CPU this technology.
system SPATIAL AUDIO
PIXEL PERSISTENCE VR Additionally, in order to improve the visual experience the whole sub-
HEAD TRACKING Headset system, starting from input control to image display needs to be optimised
BODY TRACKING for minimal latency and maximum throughput.
HAPTICS
Other
INPUT Peripherals This includes, rendering in 2D, 3D and immersing displays with texture
based rendering.
41
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
CPU and GPU Performance over time
The adjacent CPU/GPU graph suggests, using Moore’s Law,
coupled with Nanotechnology together with advancements
in parallel computation has boosted the overall performance
of GPU in relation to graphic intensive computation.
Since 1997, each order of magnitude only took three years (5.7
orders of magnitude in 18 years).
42
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Key outcome of the GPU and Battery case
GPU horsepower case Battery case
1. There are cards that already can satisfy minimum
1. Batteries will still be the problematic part of
requirements for realistic immersive VR systems. But
the equation. The energy density/weight
they will work only with high-end laptops and PC for
and energy density/volume ratios are
the next 5 years.
progressing not so fast as energy
2. According to current trends the GPU horsepower will
density/cost ratio. It’s very critical issue for
be increasing, but the power and weight will also be
mobile VR systems and will not be solved
increasing – 250 Wt - $1000 price and 1 kg weight are
for the next 5-10 years according to current
limiting factors for mobile VR.
trends in technology.
3. This leads us to the total cost of VR system for the
end user (gaming market). It will consist of laptop +
high-end GPU + HMD + content = $2000 + $1000 +
$ 1500 + $1000 = $5500 per year per 10 pieces of
content. A bit expensive, isn’t it? The only way I see is rethinking of
4. Mobile VR will be developing with low-end graphics computing architectures. What AMD,
and will not give opportunity to be totally immersed. Nvidia, and Intel already are doing….
5. Mixed systems will be developing where
computational power will be in the backpack of the
gamer
43
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Progress within VR & AR
Recently, AMD has launched Project: LiquidVR® In other developing events, Intel has partnered
dedicated to VR and AR technology. According to with Oculus Rift to form VR alliance in
AMD, GPU software and hardware sub-systems to association with Altspace VR to establish a
tackle the common issues and pitfalls of achieving common platform for better user experience.
presence, such as reducing motion-to-photon latency
to less than 10 milliseconds. Additionally, 6th generation Intel processors
have incorporated Intel RealSense®
“LiquidVR is a silky smooth player” –
Technology which is a gesture based Human-
The Guardian
Computer Interaction model to process and
compute AR/VR information.
44
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Challenges and barriers of AR
46
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
What are key areas for innovation in AR & VR?
VR will kick off graphics into hyperdrive :
• 90 hz stereo, Can't miss frames
• 110 fov
• Resolution of 10k x 10k
• High rendering quality
• New rendering techniques
• GPU performance 2 to 3 orders of magnitude
• Transmission links
• Display ( 100 megapixels and more, thousands of pixels per inch, HDR)
• Rethinking art and animation technology
• Optics
• Rendering engines and content creation platforms
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
What are key areas for innovation in AR & VR?
● Virtual reality will be most popular in the gaming market, because of the high level of immersion
possible in VR, as well as the high demand for interactive experiences.
● Augmented reality will be most successful in the enterprise market, for applications in logistics,
engineering, and automotive.
● Applications such as education, travel, and design are served well by both AR and VR, and success
depends more on specific needs than general application.
● New content mediums, such as interactive VR video, are becoming more popular and catching the
attention of content creators as a potentially viable medium. Existing software solutions, such as CAD
software, can be outfitted to support AR and VR devices, and open a new approach to using the
software.
Scenario 1.
• VR will be the first in mass adoption.
• The roadmap for VR is likely to be pretty straightforward: gaming market,
entertainment market.
Scenario 2.
• AR will be second and after 2020.
• Industrial applications of AR find it’s slow way to the market overcoming
technical & organizational barriers.
• Information glasses succeed.
Scenario 3.
• Console AR revolutionizes living-room gaming.
• Next step – whole new computing platform. Laptops and PCs are replaced
by HMD.
57
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru
Our Team 58
Vasily Ryzhonkov,
Director of Mobile Tech Center,
Skolkovo Foundation
vryzhonkov@sk.ru
http://mobile.sk.ru
Joshua Levy
Intern at Mobile Tech
Center, Skolkovo
Foundation
58
Vasily Ryzhonkov : vryzhonkov@sk.ru : +7 (916) 004 03 15 : http://mobile.sk.ru